lmxopcua/docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md.tasks.json

{
  "planPath": "docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md",
  "note": "Per-cluster mesh Phase 3. Decisions pre-made by the user: gRPC fetch RPC (NOT token-gated HTTP); shared node key (NOT per-deployment token, so no migration and DispatchDeployment is UNCHANGED); config-flagged dark switch ConfigSource:Mode=Direct default; both pair nodes fetch (no Primary gating). Five recon facts + the phase-boundary rule (Phase 3 = config READS only; NodeDeploymentState write / DbHealthProbe / EfAlarmConditionStateStore stay for Phase 4) are in the plan header. RevisionHash == SHA-256(blob) is load-bearing for fetch verification.",
  "tasks": [
    {
      "id": 0,
      "subject": "Task 0: ConfigSource/ConfigServe options + validator + registration",
      "status": "completed",
      "classification": "small",
      "parallelizableWith": [
        1
      ],
      "commit": "9d1e60c0",
      "note": "One sabotage (accept empty endpoints) reddened its test; file was untracked so restored by hand not git checkout."
    },
    {
      "id": 1,
      "subject": "Task 1: deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen",
      "status": "completed",
      "classification": "standard",
      "parallelizableWith": [
        0
      ],
      "commit": "62e8f54f",
      "note": "First locally-compiled proto in the repo. Grpc.Tools 2.76.0 pinned (aligned with the other Grpc 2.76.0 refs); GrpcServices=Both. Commons builds clean, codegen verified."
    },
    {
      "id": 2,
      "subject": "Task 2: central DeploymentArtifactService \u2014 stream sealed artifact, NotFound-hide the rest",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        1
      ],
      "parallelizableWith": [
        4
      ],
      "commit": "7d7de482",
      "note": "Impl RENAMED to DeploymentArtifactGrpcService \u2014 the generated container class is itself named DeploymentArtifactService and its nested base collided. Aliased the base. Sabotage dropping the sealed+empty guard reddened 2 tests."
    },
    {
      "id": 3,
      "subject": "Task 3: config-serve auth interceptor + dedicated h2c listener wiring",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        0,
        2
      ],
      "commit": "4d88f67c",
      "note": "Merged the LocalDb-sync + ConfigServe listener blocks into ONE that re-applies the existing HTTP surface exactly once and adds both dedicated h2c ports; a fused admin+driver node double-binding the HTTP port throws 'address already in use' (sabotage-proven). AddGrpc moved out of the hasDriver block to hasDriver||hasAdmin with both path-scoped interceptors. StatusCode.Unauthenticated (LocalDb uses PermissionDenied). Coexistence proven by a minimal-WebApplication shape test, mirroring LocalDbSyncListenerTests; real end-to-end fetch is Task 8."
    },
    {
      "id": 4,
      "subject": "Task 4: node gRPC artifact fetcher \u2014 SHA-verified reassembly, endpoint failover, null-on-any-failure",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        1
      ],
      "parallelizableWith": [
        2
      ],
      "commit": "7a8fb5f1",
      "note": "Uniform invariant: return the first endpoint's bytes that reassemble AND verify (SHA-256 lowercase-hex == dispatched RevisionHash); else null. RpcException / empty-stream / hash-mismatch all -> try-next, never throw. Func<endpoint,client> seam keeps gRPC out of unit tests (fake subclasses the generated client's protected ctor). Namespace is .DeploymentCache NOT .Deployment (else it shadows Configuration.Entities.Deployment across the whole test assembly via enclosing-namespace lookup) \u2014 same trap LocalDbDeploymentArtifactCache already dodged. Added Grpc.Net.Client to Runtime.csproj. xUnit v2 here: CancellationToken.None, not TestContext.Current."
    },
    {
      "id": 5,
      "subject": "Task 5: FetchAndCache apply path \u2014 fetch->cache->apply-from-bytes, null=apply-failure",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        4
      ],
      "commit": "648f173b",
      "note": "DispatchDeployment (rev differs) in FetchAndCache -> BeginFetchAndCacheApply PipeTo(Self) FetchedForApply(bytes?), never a blocking await in a receive. HandleFetchedForApply applies synchronously mirroring ApplyAndAck: null bytes = FAILURE (keep last-known-good, don't advance rev, ack Failed); good bytes reconcile+rebuild+push FROM the bytes (OpcUaPublishActor never reads SQL) then cache. A faulted fetch task maps to null via PipeTo failure:. Extracted ReconcileDriversFromBlob (Direct/FetchAndCache/boot all share it; #485 empty guard moved in); ApplyCachedArtifact now reuses it. Props gains fetchAndCacheMode+artifactFetcher LAST. Idempotent re-dispatch is the existing _currentRevision short-circuit (no separate cache-contains check added; peer-replication-missed-fetch is covered by Task 6 boot + StoreAsync idempotency). Sabotages: advance-rev-on-null reddens test b; drop _currentRevision guard reddens test c."
    },
    {
      "id": 6,
      "subject": "Task 6: FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        5
      ],
      "commit": "6d004738",
      "note": "Bootstrap() branches to BootstrapFromCache() in FetchAndCache: GetCurrentUnkeyedAsync -> set _currentRevision -> ApplyCachedArtifact (no SQL, no re-ack). _isRunningFromCache STAYS false (normal op, not degraded). Empty/unreadable cache -> Steady-no-revision (first dispatch fetches), NEVER Stale (Stale = 'SQL down, retry'; FetchAndCache has no config SQL read to recover). TryRecoverFromStale gains a defensive FetchAndCache early-return (unreachable anyway \u2014 retry-db timer only starts in Stale). KEY test lever: UpsertNodeDeploymentState SWALLOWS db failures, so a FetchAndCache dispatch runs end-to-end under a ThrowingDbFactory \u2014 the ack proves Steady-not-Stale AND no-SQL-at-boot in one shot. NodeDiagnosticsSnapshot lives in Commons.Interfaces not Messages.Fleet."
    },
    {
      "id": 7,
      "subject": "Task 7: #485 empty/corrupt/mismatch coverage on the FetchAndCache paths",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        6
      ],
      "commit": "fc568ae0",
      "note": "Regression net, no new mechanism \u2014 all green (no gaps to guard). Pins the NEGATIVE on failure paths: (1) null fetch mid-steady-state -> no RebuildAddressSpace for the failed rev, revision + served state kept (publishProbe.ExpectNoMsg is the discriminator); (2) zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485 guard (independent of the fetcher null contract); (3) corrupt cache at boot = GetCurrentUnkeyedAsync null -> Steady-no-revision, no rebuild at boot (no partial apply). Removing the #485 empty guard reddens the zero-length test. Corrupt-cache uses a stub-returns-null (the real ReassembleAsync-null-on-corrupt is LocalDbDeploymentArtifactCacheTests' job)."
    },
    {
      "id": 8,
      "subject": "Task 8: real two-Host gRPC artifact-fetch boundary test + wrong-key falsifiability control",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        3,
        4
      ],
      "parallelizableWith": [
        9
      ],
      "commit": "ffbcaa93",
      "note": "Minimal real Host (WebApplication + h2c-only Kestrel + real ConfigServeAuthInterceptor + real DeploymentArtifactGrpcService + in-memory EF sealed deployment) driven by the REAL GrpcDeploymentArtifactFetcher. 4 green: right-key >256KB reassembles byte-equal + verifies; wrong-key null WHILE right-key succeeds on the SAME server (falsifiability control); unknown-id null; [dead-port,real-port] failover. h2c GrpcChannel.ForAddress(http://) works out of the box in .NET 10 \u2014 no AppContext switch needed. async can't have out params -> StartServerAsync returns a tuple. Tagged [Trait Category=ArtifactBoundary]."
    },
    {
      "id": 9,
      "subject": "Task 9: docker-dev rig config + docs + program-plan flip",
      "status": "completed",
      "classification": "small",
      "blockedBy": [
        5
      ],
      "parallelizableWith": [
        8
      ],
      "commit": "cf6110d0",
      "note": "Centrals: ConfigServe :4055 + committed dev key, ConfigSource:Mode=Direct (they never fetch). Sites: ConfigSource endpoints (both centrals) + matching key, Mode=${OTOPCUA_CONFIG_MODE:-Direct}. One env var flips ONLY the sites because centrals hard-code Direct. Docs: Configuration.md ConfigSource/ConfigServe section, Redundancy.md out-of-band note, program plan Phase 3 CODE-COMPLETE + tracking row, CLAUDE.md Config-source section."
    },
    {
      "id": 10,
      "subject": "Task 10: live gate on docker-dev (site nodes -> FetchAndCache; step 4 = stop central SQL mid-fetch = the headline gate)",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        8,
        9
      ],
      "commit": "(gate doc) 2026-07-22-mesh-phase3-live-gate.md",
      "note": "PASSED \u2014 and CAUGHT A REAL DEFECT every offline test missed: DeploymentArtifactGrpcService gated on Status==Sealed, but central seals only AFTER all nodes ack, and a FetchAndCache node can't ack until it fetches => seal-needs-ack->ack-needs-fetch->fetch-needs-sealed DEADLOCK. Fetch reached central + passed the interceptor, got clean NotFound, #485 kept last-known-good, deploy never sealed. Fixed by dropping the Sealed gate (serve any non-empty blob; Direct reads the same AwaitingApplyAcks row). Task 2 test flipped to non-sealed-with-blob-is-served. After fix: fetch+verify+apply+seal green 6/6. HEADLINE step 4 PASS: SQL stopped, sites keep serving, restarted site boots last-applied config from the LocalDb pointer, no crash, never Stale. Findings: Kestrel takeover+AdminUI coexist live on central (:9000+:4055); transient Direct SQL-reconnect blip on central-1 is #486 not a regression; failover to central-2 works; deploy POST needs Content-Type:application/json (else 404). Build: Dockerfile build stage pinned to linux/amd64 \u2014 Grpc.Tools linux_arm64 protoc segfaults (139) on Apple-Silicon Docker; portable-IL publish keeps runtime native arm64. Steps 8/9 folded into Task7/Task8/Task3 real coverage rather than slow emulated recreate."
    }
  ],
  "lastUpdated": "2026-07-23T01:00:00Z"
}