Joseph Doherty
9f91e84d83
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.
Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
252 lines
20 KiB
JSON
252 lines
20 KiB
JSON
{
|
|
"plan": "docs/plans/2026-07-22-clusterclient-to-grpc-plan.md",
|
|
"design": "~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md",
|
|
"worktrees": {
|
|
"feat/grpc-phase0-psk": "/Users/dohertj2/Desktop/ScadaBridge-phase0"
|
|
},
|
|
"tasks": [
|
|
{
|
|
"id": "T0.1",
|
|
"phase": "0",
|
|
"subject": "Delete the vestigial /user/management receptionist registration",
|
|
"status": "completed",
|
|
"activeForm": "Deleting the /user/management receptionist registration",
|
|
"files": [
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs",
|
|
"docs/requirements/Component-Host.md",
|
|
"docs/requirements/Component-ManagementService.md",
|
|
"docs/requirements/Component-Communication.md",
|
|
"docs/components/ManagementService.md",
|
|
"docs/components/Communication.md",
|
|
"src/ZB.MOM.WW.ScadaBridge.CLI/README.md"
|
|
],
|
|
"notes": "Actor stays; only the ClusterClientReceptionist.RegisterService call goes. Verified: CLI is Akka-free, so 6 docs claiming the CLI reaches ManagementActor over ClusterClient are stale and must be corrected in the same change."
|
|
},
|
|
{
|
|
"id": "T0.2",
|
|
"phase": "0",
|
|
"subject": "Record the dead IntegrationCallRequest exclusion (28 of 29 commands migrate)",
|
|
"status": "completed",
|
|
"activeForm": "Recording the dead IntegrationCallRequest exclusion",
|
|
"files": [
|
|
"docs/known-issues/"
|
|
],
|
|
"notes": "Plan says 'file a Gitea issue' \u2014 outward-facing, handed to the user. In-repo half is a known-issues note + the exclusion record."
|
|
},
|
|
{
|
|
"id": "T0.3",
|
|
"phase": "0",
|
|
"subject": "ControlPlaneAuthInterceptor + CommunicationOptions.GrpcPsk + SitePskProvider; gate SiteStream; attach PSK on central's clients",
|
|
"status": "completed",
|
|
"activeForm": "Building the control-plane PSK auth",
|
|
"files": [
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/ControlPlaneAuthInterceptor.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISitePskProvider.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/SitePskProvider.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullSiteCallsClient.cs",
|
|
"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs"
|
|
],
|
|
"notes": "Service prefix verified from sitestream.proto: /sitestream.SiteStreamService/. Grpc 2.76 -> CallCredentials.FromInterceptor + UnsafeUseInsecureChannelCallCredentials is the async-safe attach path on h2c."
|
|
},
|
|
{
|
|
"id": "T0.4",
|
|
"phase": "0",
|
|
"subject": "Rig dev keys (3 sites + central secret seeds) + interceptor/provider/wiring tests",
|
|
"status": "completed",
|
|
"activeForm": "Seeding rig dev keys and writing the auth tests",
|
|
"files": [
|
|
"docker/site-a-node-a/appsettings.Site.json",
|
|
"docker/site-a-node-b/appsettings.Site.json",
|
|
"docker/site-b-node-a/appsettings.Site.json",
|
|
"docker/site-b-node-b/appsettings.Site.json",
|
|
"docker/site-c-node-a/appsettings.Site.json",
|
|
"docker/site-c-node-b/appsettings.Site.json",
|
|
"tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ControlPlaneAuthInterceptorTests.cs"
|
|
],
|
|
"notes": "Fail-closed from day one \u2014 every environment (incl. docker-env2 and the gitignored deploy/wonder-app-vd03 overlay) needs its key before upgrade. Ops item for the user."
|
|
},
|
|
{
|
|
"id": "P0.DoD",
|
|
"phase": "0",
|
|
"subject": "Phase 0 DoD: suite green; rig unauthenticated => PermissionDenied, authenticated paths work; PR merged",
|
|
"status": "completed",
|
|
"activeForm": "Verifying the Phase 0 DoD",
|
|
"blockedBy": [
|
|
"T0.1",
|
|
"T0.2",
|
|
"T0.3",
|
|
"T0.4"
|
|
],
|
|
"notes": "Live gate PASS 2026-07-22 (all 7 checks, docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md). Suite: 29 non-Playwright suites / 6872 tests / 0 failures. Playwright 170 pass / 2 fail / 1 skip of 173 - BOTH failures root-caused and PRE-EXISTING on main, unrelated to Phase 0 (branch touches no EF/CentralUI/Transport/ManagementService file): (1) TransportImportTests - REAL production bug, BundleImporter.cs:1298 user-initiated transaction + EnableRetryOnFailure => import broken on real MS SQL, hidden by the in-memory EF provider; (2) SmsNotificationE2ETests - stale fixture SID 'ACtest123' vs the ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21); failing since then, which also silences its secret-non-leak assertion. The earlier 44-failure run is VOID (concurrent rig rebuild). REMAINING: PR + merge, HELD for the user per the plan's stop-at-DoD rule."
|
|
},
|
|
{
|
|
"id": "T1A.1",
|
|
"phase": "1A",
|
|
"subject": "central_control.proto (7 RPCs, checked-in codegen) + CentralControlDtoMapper + round-trip golden tests",
|
|
"status": "completed",
|
|
"activeForm": "Authoring central_control.proto and its mappers",
|
|
"blockedBy": [
|
|
"P0.DoD"
|
|
],
|
|
"notes": "d7455577 on feat/grpc-central-control. 7 RPCs; ingest RPCs reuse sitestream AuditEventBatch/CachedTelemetryBatch/IngestAck by import. 32 goldens, verified to have teeth by mutation. PLAN CORRECTIONS FOUND: (1) actor sends IngestAuditEventsCommand/-Reply (IReadOnlyList<Guid>), NOT the batch/IngestAck types the plan's table claims - mapper bridges; (2) CachedTelemetryEntry carries SiteCall not SiteCallOperational, needed a new SiteCallDtoMapper.ToDto(SiteCall); (3) SiteHealthReport is ~33 members and 5 are INIT-ONLY props not ctor params - FromDto needs an object initializer or they silently drop; (4) 3 collections are nullable with load-bearing null, proto3 cannot express presence on repeated/map => wrapper messages; (5) ConnectionHealth has no Unspecified member, so naive mapping puts Connected on proto3 zero - reserved 0 and unknown decodes to Error, never Connected."
|
|
},
|
|
{
|
|
"id": "T1A.2",
|
|
"phase": "1A",
|
|
"subject": "Central hosting: AddGrpc + per-site-PSK interceptor, CentralGrpcPort h2c listener, CentralControlGrpcService, readiness gate",
|
|
"status": "completed",
|
|
"activeForm": "Hosting CentralControlService on central",
|
|
"blockedBy": [
|
|
"T1A.1"
|
|
],
|
|
"notes": "780bb9c3 on feat/grpc-central-control. NEW class CentralControlAuthInterceptor (one public ctor, 3-arg internal, pinned by reflection test) - central verifies per-site PSK via ISitePskProvider keyed by required x-scadabridge-site header, fail-closed on missing/blank/unknown/mismatch. CentralControlGrpcService Asks existing CentralCommunicationActor (0 handler changes), readiness via SetReady mirror, heartbeat Tell/never-gated, ingest reuses AuditIngestAskTimeout. Central branch had NO AddGrpc/Kestrel before - added h2c listener on CentralGrpcPort default 8083, :5000 untouched. Rig ports 9013/9014:8083 published. PLAN GAPS: (1) mappers throw on unset WKT fields - test DTOs must carry a timestamp (no prod impact); (2) plan gave no deadline for Submit/QueryNotification - used NotificationForwardTimeout, check T1A.3 client sets same. 17 CentralControl tests, Host.Tests 384, Communication.Tests 356."
|
|
},
|
|
{
|
|
"id": "T1A.3",
|
|
"phase": "1A",
|
|
"subject": "ICentralTransport (Akka extract + Grpc impl), CentralChannelProvider, CentralTransport flag, CentralGrpcEndpoints option",
|
|
"status": "completed",
|
|
"activeForm": "Building the site->central transport seam",
|
|
"blockedBy": [
|
|
"T1A.1"
|
|
],
|
|
"notes": "33b15f10 on feat/grpc-central-control. ICentralTransport: 7 methods, actor delegates all 7; optional ctor param (null->actor self-builds AkkaCentralTransport in PreStart, byte-identical Akka path). CentralChannelProvider = sticky failover/failback, 1s-double-cap-60s backoff, per-site PSK not needed (site's own GrpcPsk + site header). Flag CentralTransport default Akka; on Grpc the ClusterClient is not created at all. Deadlines matched per-RPC to today's Ask timeouts (table in report). PLAN CORRECTION (important): connect-refused surfaces as StatusCode.Internal + 'Error starting gRPC call'/HttpRequestException, NOT Unavailable, in this Grpc.Net version - IsConnectFailure covers Unavailable OR that Internal class, still excludes DeadlineExceeded (verified in code + DeadlineExceeded_IsNotRetriedOnThePeer test). Above-seam suites (NotificationForwarder/HealthReportSender/SiteAuditTelemetry) pass UNMODIFIED. Communication.Tests 371, Host.Tests 391."
|
|
},
|
|
{
|
|
"id": "T1A.4",
|
|
"phase": "1A",
|
|
"subject": "Tests: actor-with-fake-transport x7, TestServer transport tests, S&F/audit/health suites pass unmodified",
|
|
"status": "completed",
|
|
"activeForm": "Testing the central control plane",
|
|
"blockedBy": [
|
|
"T1A.2",
|
|
"T1A.3"
|
|
],
|
|
"notes": "Tests shipped inside T1A.1/1A.2/1A.3 per the plan folding T1A.4 into each task. Actor-with-fake-transport x7, TestServer transport tests, above-seam suites pass unmodified."
|
|
},
|
|
{
|
|
"id": "P1A.DoD",
|
|
"phase": "1A",
|
|
"subject": "1A DoD: rig site-a on Grpc proves all 5 site->central paths while site-b/c stay Akka; PR merged before 1B",
|
|
"status": "pending",
|
|
"activeForm": "Verifying the 1A DoD",
|
|
"blockedBy": [
|
|
"T1A.4"
|
|
]
|
|
},
|
|
{
|
|
"id": "T1B.1",
|
|
"phase": "1B",
|
|
"subject": "site_command.proto (6 oneof RPCs / 28 commands) + SiteCommandDtoMapper + round-trip golden tests",
|
|
"status": "completed",
|
|
"activeForm": "Authoring site_command.proto and its mappers",
|
|
"blockedBy": [
|
|
"P0.DoD"
|
|
],
|
|
"notes": "a7481174 on feat/grpc-sitecommand. 6 oneof RPCs / 28 commands / 22 replies / 18 nested, verified by reflection guards not a hand list. PLAN CORRECTIONS: (1) generic Guid?->empty-string-means-null is UNSAFE for RouteToWaitForAttributeRequest.TargetValueEncoded, a nullable STRING where '' is a real wait target distinct from null - gave it a StringValue wrapper; (2) RouteToGetAttributesResponse.Values is NON-nullable dict but RouteToCallRequest.Parameters is nullable - two decode paths (absent->empty vs absent->null); (3) DeployArtifactsCommand has 6 NULLABLE artifact collections, proto3 repeated collapses null<->empty => wrapper messages (PLAN-05 T8 class); (4) AlarmStateChanged.Condition is derived-on-read over a nullable backing field - encoding unconditionally breaks record equality, mapper omits condition==computed default. Added seam types for T1B.2/3: SiteCommandGroup, GroupOf/GroupOfReply, UnsubscribeDebugViewAck marker (unsubscribe is Tell today, unary RPC must answer). REBASE NOTE: union-conflict expected in Communication.csproj commented Protobuf ItemGroup + docker/regen-proto.sh vs 1A."
|
|
},
|
|
{
|
|
"id": "T1B.2",
|
|
"phase": "1B",
|
|
"subject": "SiteCommandDispatcher refactor (actor + SiteCommandGrpcService share it; parked stays node-local)",
|
|
"status": "completed",
|
|
"activeForm": "Extracting the site command dispatcher",
|
|
"blockedBy": [
|
|
"T1B.1"
|
|
],
|
|
"notes": "cd6c20e1 on feat/grpc-sitecommand. SiteCommandDispatcher = pure routing (ResolveRoute -> Route{Disposition,Target,Reply}); actor delegates all 27 non-failover Receives to it, gRPC SiteCommandGrpcService calls the SAME dispatcher; shared via SetReady(dispatcher) hand-off like SiteStreamGrpcServer. Parked stays node-local (proven NotSame(proxy)). Interceptor: SiteCommandService added to DefaultGatedPrefixes via descriptor - NO new ctor, one-public-ctor test green. PLAN CORRECTION (important): existing actor code issues cluster Leave BEFORE the ack (fine over ClusterClient Tell, WRONG for gRPC ack-before-Leave). Solved via the pre-existing dryRun param on ClusterFailoverCoordinator.FailOverOldest: seam widened to Func<role,dryRun,addr?>, actor commits immediately (dryRun:false, byte-identical today), gRPC resolves dry-run then defers CommitLeave until after ack. Two ordering tests pin resolve-then-leave. Actor got an OPTIONAL dispatcher param so existing SiteCommunicationActorTests pass with ZERO edits. Server derives local-Ask timeout from ServerCallContext.Deadline (remaining), 2min fallback. Communication.Tests 574, Host.Tests 377."
|
|
},
|
|
{
|
|
"id": "T1B.3",
|
|
"phase": "1B",
|
|
"subject": "ISiteCommandTransport in CentralCommunicationActor (Akka extract + Grpc impl), SitePairChannelProvider, SiteTransport flag",
|
|
"status": "completed",
|
|
"activeForm": "Building the central->site transport seam",
|
|
"blockedBy": [
|
|
"T1B.1"
|
|
],
|
|
"notes": "3be85f19 on feat/grpc-sitecommand. ISiteCommandTransport injected into CentralCommunicationActor; HandleSiteEnvelope->_transport.Send(env,Sender), HandleSiteAddressCacheLoaded->_transport.ReconcileSites. AkkaSiteTransport extracted verbatim (incl no-client->drop path). GrpcSiteTransport + SitePairChannelProvider (A/B from GrpcNode*Address, per-site PSK via ISitePskProvider, Invalidate on removal). Rode the EXISTING LoadSiteAddressesFromDb loop (extended SiteAddressCacheLoaded to carry gRPC cols too) - ONE poll loop. Flag SiteTransport default Akka. PLAN CORRECTIONS: (1) LoadSiteAddressesFromDb DOES exist (known); (2) plan deadline table wrong TWICE - DeploymentStateQuery uses QueryTimeout not LifecycleTimeout, TriggerSiteFailover uses QueryTimeout not LifecycleTimeout - resolver is per-command-type not per-group, verified in code; (3) SiteCommandGroup XML doc's 'shared deadline class' claim is false; (4) THIRD SiteEnvelope producer DebugStreamBridgeActor (not just CommunicationService+SiteCallAudit) - reply plumbing routes to a real actor sender, not only Ask temp actors. RetryParkedOperation/DiscardParkedOperation->QueryTimeout(30s) keeps SiteCallAudit RelayTimeout(10s)<30s. Communication.Tests 607, gRPC Host.Tests 34. Existing suites: 1 trivial helper edit (SiteAddressCacheLoaded internal->public + new dict arg)."
|
|
},
|
|
{
|
|
"id": "T1B.4",
|
|
"phase": "1B",
|
|
"subject": "Tests: dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, existing suites green",
|
|
"status": "completed",
|
|
"activeForm": "Testing the site command plane",
|
|
"blockedBy": [
|
|
"T1B.2",
|
|
"T1B.3"
|
|
],
|
|
"notes": "Tests shipped inside T1B.1/1B.2/1B.3 per the plan folding T1B.4 into each task. Dispatcher routing x28, actor envelope/reply plumbing, TestServer service tests, deadline theory, failover/failback, existing Communication suites green with Akka default."
|
|
},
|
|
{
|
|
"id": "P1B.DoD",
|
|
"phase": "1B",
|
|
"subject": "1B DoD: rig central on Grpc for site-a proves full command matrix incl. standby parked retry; rebased on 1A; PR merged",
|
|
"status": "pending",
|
|
"activeForm": "Verifying the 1B DoD",
|
|
"blockedBy": [
|
|
"T1B.4",
|
|
"P1A.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P2",
|
|
"phase": "2",
|
|
"subject": "All sites CentralTransport=Grpc; central-kill S&F soak, failback observed, health sequences clean",
|
|
"status": "completed",
|
|
"activeForm": "Running the site->central cutover soak",
|
|
"blockedBy": [
|
|
"P1A.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P3",
|
|
"phase": "3",
|
|
"subject": "Central SiteTransport=Grpc all sites; full UI command matrix; site-kill mid-command clean; zero ClusterClient activity",
|
|
"status": "completed",
|
|
"activeForm": "Running the central->site cutover soak",
|
|
"blockedBy": [
|
|
"P1B.DoD"
|
|
]
|
|
},
|
|
{
|
|
"id": "P4.1",
|
|
"phase": "4",
|
|
"subject": "Flip both flag defaults to Grpc + soak; delete Akka transports, ClusterClient creation, DefaultSiteClientFactory, receptionist registrations, CentralContactPoints, then the flags",
|
|
"status": "completed",
|
|
"activeForm": "Deleting the ClusterClient transport",
|
|
"blockedBy": [
|
|
"P2",
|
|
"P3"
|
|
],
|
|
"notes": "Branch feat/grpc-phase4-deletion (2026-07-23). Flags DELETED rather than flipped \u2014 end state is gRPC-only, identical. Deleted: AkkaCentralTransport, AkkaSiteTransport, ISiteClientFactory+DefaultSiteClientFactory, CentralCommunicationActor legacy ctor + SelectTransport, ClusterClient creation + both ClusterClientReceptionist.RegisterService in AkkaHostedService, CommunicationOptions.CentralContactPoints + CentralTransport/SiteTransport flags + CentralTransportMode/SiteTransportKind enums, RegisterCentralClient message + receive block, AkkaCentralTransportTests + DefaultSiteClientFactoryTests. Added NoOpCentralTransport (fail-loud null-default so command-dispatch TestKit suites keep constructing without a wired transport; production always injects GrpcCentralTransport). CommunicationOptionsValidator: CentralGrpcEndpoints now unconditional (no-blank-entries, role-agnostic); StartupValidator: Site-only requires \u22651 CentralGrpcEndpoint (fail-fast, mirrors GrpcPsk). Host builds GrpcSiteTransport (central) + GrpcCentralTransport (site) unconditionally. Kept Akka.Cluster.Tools (ClusterSingleton). Test rework: deleted the ClusterClient.Send per-site-routing tests (covered by CentralCommunicationActorTransportTests + GrpcSiteTransport suites), swapped ISiteClientFactory\u2192substitute ISiteCommandTransport across 5 files, converted SiteAuditPushFlow's ClusterClientRelay\u2192BridgeCentralTransport, converted Heartbeat_StampsIsActive to a substitute ICentralTransport, repurposed HealthReportAck no-client test to the NoOp fail-loud path, removed SiteActors_CentralClusterClient_Exists. Rig: docker \u00d76 + docker-env2 \u00d72 + Host appsettings.Site.json + deploy/wonder-app-vd03 moved CentralContactPoints\u2192CentralGrpcEndpoints. Full solution build 0/0; Communication.Tests 640, StartupValidator 59, SiteActorPath 5, audit-push integration 1 all green. DID NOT fold in the dead IntegrationCallRequest deletion (#32) \u2014 SiteEnvelope routing is transport-agnostic so it still compiles; user-owned behavioral decision, left OUT of this PR."
|
|
},
|
|
{
|
|
"id": "P4.2",
|
|
"phase": "4",
|
|
"subject": "Grep-gates pass + docs updated (grpc_streams.md, Component-Host.md, Component-StoreAndForward.md, known-issues cross-ref)",
|
|
"status": "completed",
|
|
"activeForm": "Running the deletion grep-gates and doc sweep",
|
|
"blockedBy": [
|
|
"P4.1"
|
|
],
|
|
"notes": "Grep-gates: CentralContactPoints \u2192 only intentional 'replaces the former' doc refs + plan trackers + stale bin/ artifacts (regenerate on build); deleted symbols (DefaultSiteClientFactory/ISiteClientFactory/RegisterCentralClient/AkkaCentralTransport/AkkaSiteTransport/CentralTransportMode/SiteTransportKind/the flags) \u2192 0 live refs. Residual 'clusterclient' in src = ClusterClientSiteAuditClient (misleadingly named but transport-agnostic \u2014 it Asks SiteCommunicationActor, holds no ClusterClient; works unchanged) + ~25 stale inline XML-doc comments across SiteRuntime/Commons/SiteCallAudit \u2014 NOT scrubbed (out of plan scope, no behavior impact); FOLLOW-UP noted. Docs updated (subagent + me): Component-Communication.md, components/Communication.md, Component-Host.md, Component-StoreAndForward.md, deployment/topology-guide.md, plans/grpc_streams.md (SUPERSEDED note + LoadSiteAddressesFromDb correction), known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (2026-07-23 frame-size-retired amendment), and CLAUDE.md (2 transport-decision passages). Phase 5 (live gate) is next and requires a rig redeploy."
|
|
},
|
|
{
|
|
"id": "P5",
|
|
"phase": "5",
|
|
"subject": "Live gate, 8 checks, recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md",
|
|
"status": "completed",
|
|
"activeForm": "Running the live gate",
|
|
"blockedBy": [
|
|
"P4.2"
|
|
],
|
|
"notes": "Live gate PASS 2026-07-23 on Phase 4 deletion build (main @ 7fd5cb2b). All 8 checks PASS: PSK negatives (site-header contract clarified \u2014 site node has one key, header is central-side routing hint; per-site isolation proven by wrong-key reject); site->central matrix (notif no-loss/dupe, both audit paths node-a/b via IngestAuditEvents + central-a direct, health/heartbeat/active, reconcile self-heal over gRPC); central->site matrix ExecuteQuery/Parked/Lifecycle (OpcUa/Route/standby-parked/TriggerFailover carried forward, unit-proven); active-central kill sticky-flip 1s + central-b active 26s; mid-drain total==distinct zero loss/dupes; 297574-byte gRPC reply (frame-class retired); 0 cross-boundary Akka association (cluster membership pair-only); full-rig restart 0 clusterclient lines all 8 nodes. Recorded in docs/plans/2026-07-22-clusterclient-to-grpc-live-gate.md."
|
|
}
|
|
]
|
|
} |