Files

T

Joseph Doherty 2b3e1f1549 docs(plans): R3-01 live gate — PASS on docker-dev with DisableLogin=false (control failed first)

2026-08-20 12:42:06 -04:00

62 KiB

Raw Blame History

Design + Implementation Plan — R3-01 Mesh gRPC `AllowAnonymous`

Source report: archreview/03-server-runtime.md §S14 (2026-08-19 round-3 re-review at 12b0035a)
Round-1/2 designs consumed: none — S14 is NEW this round. No prior plan, branch or issue has ever targeted it.
Plan verified against tree at: master 7c77a8c7 (clean; code identical to 12b0035a) — every anchor re-opened and re-checked 2026-08-20
Scope: OVERALL prioritized action item #1 — ".AllowAnonymous() on every mesh gRPC map (the PSK interceptors are the gate); one rig/gate pass with DisableLogin=false; a standing CI variant of MeshGrpcBoundaryTests with UseAuthentication/UseAuthorization + the FallbackPolicy" (03/S14, Critical).
Cross-cutting requirement (OVERALL theme 2, "verification that cannot see its own blind spot"): the fix is one line per map; the plan's real deliverable is the verification that would have caught it. Two standing guards (a real-auth-stack boundary test + an endpoint-metadata census, both with falsifiability controls) and one rig change that makes the production middleware stack reachable on demand (Security__Auth__DisableLogin parameterised, default unchanged). A fix without those guards leaves the next map free to re-open the Critical.
Cross-cutting requirement (OVERALL theme 4, "one rule stated twice diverges"): the five map sites move behind one mapper (MapOtOpcUaMeshGrpcServices) that both Program.cs and the census test drive, so the guard can never be testing a hand-copied shape — which is precisely how the existing listener tests (ConfigServeListenerTests, LocalDbSyncListenerTests, MeshGrpcBoundaryTests) came to build pipelines that omit the middleware that breaks production.

Verification summary

S14 is CONFIRMED at 7c77a8c7 — not stale, not partially fixed, and every line anchor in the report is exact. One claim in the finding is over-stated and one is incomplete; both are corrected below and both change the work (they add one non-chainable map and one extra affected caller).

Report anchor	Claim	Status at `7c77a8c7`
`Security/ServiceCollectionExtensions.cs:143-148`	global `FallbackPolicy = RequireAuthenticatedUser` under the cookie scheme	Exact. File is `src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs` (note: not under `Host/`, as the report's shorthand reads). `AddAuthorization` at `:143`, `FallbackPolicy` built `:145-148` naming `CookieAuthenticationDefaults.AuthenticationScheme`.
`Security/ServiceCollectionExtensions.cs:82-98`	`DisableLogin` registers a pass-everything handler under the cookie scheme name	Exact. `authBuilder.AddScheme<AuthenticationSchemeOptions, AutoLoginAuthenticationHandler>(CookieAuthenticationDefaults.AuthenticationScheme, …)` at `:87-88`, with the "FallbackPolicy + FleetAdmin + DriverOperator … pass with all roles — zero policy changes" comment at `:84-86`. The masking mechanism is exactly as described.
`Host/Program.cs:520-523`	fallback registered on `hasAdmin` nodes only	Exact. `if (hasAdmin) { builder.Services.AddOtOpcUaAuth(builder.Configuration); … }` at `:520-523`.
`Host/Program.cs:732-733`	`UseAuthentication`/`UseAuthorization` in the pipeline	Exact, and also inside `if (hasAdmin)` (`:730-741`) — which is what makes the driver-only exemption structural rather than incidental. `UseAntiforgery()` follows at `:734` (harmless for gRPC: the antiforgery middleware only validates endpoints carrying `IAntiforgeryMetadata`).
`Host/Program.cs:724-728`	`MapStaticAssets().AllowAnonymous()` proves the author knew about the fallback	Exact. The comment at `:724-727` states the failure mode verbatim ("401s every CSS/JS request"). Two further precedents: `AdminUI/Api/DeployApiEndpoints.cs:61` (`.AllowAnonymous() // gated by the API key, not the cookie auth fallback`) and `Host/Observability/ObservabilityExtensions.cs:56` (`app.MapZbMetrics().AllowAnonymous()`). Three call sites already apply the pattern; the mesh maps are the omission.
`Host/Program.cs:749, 757, 767, 782-783`	five mesh maps, none carrying `AllowAnonymous`	Exact line-for-line, and complete: `grep -rn "MapGrpcService\|MapZbLocalDbSync" src` returns exactly these five sites and nothing else.
`MeshGrpcBoundaryTests.cs:334-346`	builds a pipeline with interceptors but no auth stack	Confirmed, minor drift — the builder block is `:333-348` (`WebApplication.CreateBuilder()` at `:333`, `AddGrpc(o => o.Interceptors.Add<MeshCommandAuthInterceptor>())` at `:342`, `MapGrpcService<NodeCommandGrpcService>()` at `:347`). No `AddAuthentication`/`AddAuthorization`/`UseAuthentication`/`UseAuthorization` anywhere in the file. Additional gap the report does not name: the class only ever builds the driver-node (`NodeCommandService`) shape — the central serve side (`DeploymentArtifactService` + `CentralCommandService`), which is the fleet-wide choke point, has no boundary test at all. `ConfigServeListenerTests.cs:33-59` and `LocalDbSyncListenerTests.cs:25-27` build Kestrel-binding-only apps and say so in their own remarks ("shaped exactly like `Program.cs`'s … block").

Which maps are actually affected — determined, not assumed

The fallback reaches an endpoint iff (a) the authorization middleware is in this node's pipeline, and (b) the endpoint carries no IAuthorizeData and no IAllowAnonymous metadata. Port is irrelevant: the dedicated h2c listeners (Program.cs:692-703) are extra Kestrel endpoints on the same WebApplication, so they share one middleware pipeline and one endpoint table. There is no RequireHost, no per-port branch and no second host anywhere in Program.cs. So:

Map	Role gate	Affected on a login-enabled node?
`DeploymentArtifactGrpcService` (`:757`)	`hasAdmin && ConfigServe:GrpcListenPort > 0`	YES — always. Admin-only map on an admin-only pipeline. Every `FetchAndCache` node's artifact fetch.
`CentralCommandGrpcService` (`:767`)	same	YES — always. Every node's `ReportApply` deploy ack, from every cluster.
`TelemetryStreamGrpcService` (`:782`)	`hasDriver && Telemetry:GrpcListenPort > 0`	YES on a fused `admin,driver` node (the docker-dev central pair, and any production fused central). NO on a driver-only node.
`NodeCommandGrpcService` (`:783`)	same	YES on a fused node / NO on driver-only — same pipeline. Central→central command dispatch is affected; central→site dispatch is not.
`MapZbLocalDbSync` (`:749`)	`hasDriver && LocalDb:SyncListenPort > 0`	Conditionally — and NOT in the current fleet. The report lists it unqualified; in fact the only nodes that set `LocalDb__SyncListenPort` are the driver-only site-a pair (`docker-dev/docker-compose.yml:657`), where no authorization middleware exists. It becomes affected the moment a fused node sets a sync port. Fixed anyway (it is one config key away from being a live outage) — and it is the one map that cannot take `.AllowAnonymous()` (see below).

Driver-only nodes: confirmed unaffected, structurally. AddOtOpcUaAuth is the only caller of AddAuthorization in the host graph (grep-verified), and it sits inside if (hasAdmin) (:520-523); UseAuthentication/UseAuthorization sit inside if (hasAdmin) (:730-741). A driver-only node therefore has no authorization middleware at all and no fallback policy to apply. The report's claim holds. (This is worth a standing test, not just a reading — a future "register auth everywhere for consistency" change would silently break every site node's telemetry and command ingress. T6.)

Correction that changes the work: MapZbLocalDbSync returns IEndpointRouteBuilder, not an IEndpointConventionBuilder, so .AllowAnonymous() cannot be chained onto it (verified by reading the method signature out of ZB.MOM.WW.LocalDb.Replication 0.2.1's metadata: MapZbLocalDbSync : returns IEndpointRouteBuilder (IEndpointRouteBuilder)). The library also marks nothing anonymous itself — the assembly contains no AllowAnonymous/Authorize metadata reference at all. That map needs a different mechanism (resolved below).

Extra blast radius the finding does not spell out (fused central, DisableLogin=false, MeshTransport:Mode=Grpc): central-1/central-2 carry ClusterNode rows with a GrpcPort (docker-dev/seed/seed-clusters.sql:71-79), and CentralCommunicationActor dials one call per enabled row, its own pair included. So the MAIN pair's own deploy dispatch 401s against its own NodeCommandService, and central's telemetry dialer 401s against its own pair's TelemetryStreamService. A login-enabled central pair therefore breaks the deploy plane for all six rig nodes, not only the four sites.

Supporting facts verified: all five interceptors are fail-closed — a blank/whitespace expected key rejects every call, and the presented token is compared with CryptographicOperations.FixedTimeEquals (ConfigServeAuthInterceptor.cs:112-141, TelemetryStreamAuthInterceptor.cs:112-145, MeshCommandAuthInterceptor.cs:125-154, CentralCommandAuthInterceptor.cs:142-190, LocalDbSyncAuthInterceptor.cs:117-147); each returns early on a non-matching method prefix, so the shared AddGrpc pipeline (Program.cs:586-602) is a pass-through for foreign paths. Program.cs:574 already states the design contract in so many words: "the interceptor is the ONLY inbound auth on any endpoint (the libraries verify nothing)". Metrics are already anonymous (ObservabilityExtensions.cs:56); MapOtOpcUaHealth delegates to MapZbHealth() (Health/HealthEndpoints.cs:104) and ZB.MOM.WW.Health 0.3.0's assembly does reference AllowAnonymous — the census in T2 settles it either way rather than trusting the string. Rig masking confirmed: Security__Auth__DisableLogin: "true" on central-1 (docker-dev/docker-compose.yml:385) and central-2 (:511); both are OTOPCUA_ROLES=admin,driver,cluster-MAIN (:298, :448).

Open design choices — resolved in this plan

Fix mechanism: .AllowAnonymous() per map, not a policy, not a scheme, not a second host. The interceptors are verified fail-closed, the codebase already applies this exact pattern at three sites for the same reason, and it is the only option that leaves the failure statuses dialers already handle untouched. Alternatives weighed in Proposed design.
LocalDbSync mechanism: a single app.MapGroup("").AllowAnonymous() scoped to that one call. The library map is not chainable (verified above). The empty prefix leaves the gRPC route patterns byte-identical; the group is deliberately not extended over the four chainable maps, so the deviation stays as small as the problem.
Extract the five maps behind MapOtOpcUaMeshGrpcServices (Host, one file) rather than testing a hand-copied shape. This is the theme-4 move and it is what makes the census a production assertion. Rejected alternative: leave the maps inline and have the census rebuild them in test code — that is the same construction that produced S14.
Two guards, not one. The census (metadata) catches a map added to the helper without the convention; a Program.cs source-scan (T7, following the in-repo idiom at tests/Drivers/Cli/…/FlushLoggingConventionTests.cs:28-45) catches a map added to Program.cs outside the helper. Neither subsumes the other.
Assert the interceptor's detail string, never the status alone. ConfigServeAuthInterceptor rejects with StatusCode.Unauthenticated (:127), and Grpc.Net.Client maps an HTTP 401 to Unauthenticated as well — so "no-PSK call fails with Unauthenticated" is vacuous on the artifact service: it passes identically before and after the fix. The interceptor's messages are the discriminator ("Config-serve authentication failed.", "Telemetry stream authentication failed.", "Node command authentication failed.").
The load-bearing assertion is the ALLOW. A blocked call and a refused call both "fail"; only a correctly-keyed call succeeding through the real auth stack distinguishes the fixed build. Every deny assertion in T4/T5 is paired with the positive.
Rig: parameterise, don't flip. Security__Auth__DisableLogin: "${OTOPCUA_DISABLE_LOGIN:-true}" on the central pair — default behaviour byte-identical, and the production-faithful mode becomes a documented one-variable flip instead of an edit-and-remember-to-revert. Rejected: making false the rig default (every future rig session then needs an LDAP login through a round-robin :9200, and the shared GLAuth becomes a hard dependency of the whole rig).
No deploy-plane retry/queueing work here. Mesh dispatch is fire-and-forget by design (CLAUDE.md §Mesh command transport); S14 is an authorization defect, not a delivery-semantics one. Out of scope.

Finding 1 — 03/S14: The AdminUI cookie `FallbackPolicy` blocks every mesh gRPC ingress on a login-enabled admin node (Critical)

Restatement

AddOtOpcUaAuth installs a global FallbackPolicy = RequireAuthenticatedUser bound to the cookie scheme (Security/ServiceCollectionExtensions.cs:143-148), on every hasAdmin node (Program.cs:520-523), with UseAuthentication/UseAuthorization in the pipeline (:732-733). ASP.NET Core's authorization middleware applies the fallback to every endpoint that carries no authorization metadata — gRPC endpoints included — and none of the five mesh maps carries .AllowAnonymous() (:749, :757, :767, :782-783). The PSK bearer is not an ASP.NET authentication scheme, and gRPC interceptors run inside the endpoint invocation, i.e. strictly after the authorization middleware, so a correctly-keyed node call is challenged by the cookie handler before the interceptor's verdict can matter.

On a production (login-enabled) central this is a fleet-wide deploy-plane outage: every FetchAndCache node's artifact fetch fails on every endpoint (→ the apply fails fail-closed to last-known-good per #485 — loud, not corrupting), every gRPC ReportApply ack is dropped (→ deployments seal TimedOut), and on a fused central the pair's own command dispatch and telemetry ingress fail too. Nothing caught it because the docker-dev rig runs DisableLogin=true, which registers an always-succeeding handler under the cookie scheme name (:82-98) so the FallbackPolicy passes — i.e. every live gate of the entire seven-phase mesh program ran with the one middleware that breaks production neutralized — and because MeshGrpcBoundaryTests builds its pipelines without any auth stack (:333-348).

Verification

Confirmed exact at 7c77a8c7 (see the summary table for line-by-line). Three refinements found while verifying:

The set of affected maps is role-shaped, not uniform: two are always affected on a login-enabled node (DeploymentArtifactGrpcService, CentralCommandGrpcService); two are affected only on a fused admin,driver node (TelemetryStreamGrpcService, NodeCommandGrpcService); one (MapZbLocalDbSync) is affected only on a fused node that also sets a sync port — a shape no node in the fleet currently has. Driver-only site nodes have no authorization middleware at all.
MapZbLocalDbSync is not chainable (returns IEndpointRouteBuilder), so the "one-line-per-map fix" is one line for four maps and a group wrapper for the fifth.
The observable failure is not necessarily a 401. CookieAuthenticationEvents's default OnRedirectToLogin returns 401 only for an AJAX-shaped request (X-Requested-With); a gRPC call carries no such header, so the challenge is a 302 to /login. Whether the caller then reports Unknown ("Bad gRPC response. HTTP status code: 302") or a content-type failure depends on whether the channel's handler follows the redirect. This is why T5 asserts the interceptor's identity rather than enumerating failure statuses, and why the live gate greps for /login/302/401 together rather than "401".

Root cause

Two independent decisions, each locally correct:

A global fallback policy is the right default for a Blazor admin surface — it means a newly-added page is protected by default rather than by remembering an attribute (the same instinct as PageAuthorizationGuardTests). Its cost is that every endpoint added to that host must now opt out explicitly, and the three prior opt-outs (static assets, deploy API, metrics) were each discovered by something breaking.
The mesh gRPC endpoints authenticate in interceptors because their credential is a per-cluster PSK, not a browser identity — deliberately documented as "the interceptor is the ONLY inbound auth on any endpoint" (Program.cs:574). That statement is true of the gRPC pipeline and false of the HTTP pipeline that precedes it, and nothing in the build or the tests could hold those two sentences next to each other.

The verification gap is the same shape: every test that touches these endpoints builds the pipeline by hand, so each one reproduced the author's mental model rather than the host's actual composition.

Proposed design

Fix (5 map sites, one new file, no behavioural change to any authenticated surface):

// src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs  (new)

/// <summary>
///     Maps the mesh gRPC ingress endpoints (artifact serve, central ack, telemetry stream, node
///     command) and the LocalDb sync endpoint, each marked <c>AllowAnonymous</c>.
/// </summary>
/// <remarks>
///     <para>
///         <b>Why AllowAnonymous (archreview 03/S14).</b> On an admin-role node
///         <c>AddOtOpcUaAuth</c> installs a global <c>FallbackPolicy = RequireAuthenticatedUser</c>
///         under the cookie scheme, which the authorization middleware applies to EVERY endpoint
///         without explicit authorization metadata — gRPC included. These endpoints authenticate
///         with a per-cluster PSK bearer inside a gRPC interceptor, which runs INSIDE the endpoint
///         invocation, i.e. after that middleware. Without this opt-out a correctly-keyed node call
///         is challenged by the cookie handler before the interceptor's verdict matters: artifact
///         fetches fail, deploy acks are dropped, deployments seal TimedOut. Anonymous is safe here
///         because all five interceptors fail CLOSED — an unset key rejects every call and the
///         presented token is compared with FixedTimeEquals.
///     </para>
///     <para>
///         <b>Do not "tidy" this into Program.cs.</b> The census guard
///         (<c>MeshGrpcAnonymousMetadataTests</c>) drives THIS method, so a map that moves back
///         inline leaves the guard asserting a shape production no longer has — which is exactly
///         how S14 survived seven phases of live gates.
///     </para>
/// </remarks>
public static void MapOtOpcUaMeshGrpcServices(
    this WebApplication app,
    bool hasAdmin,
    bool hasDriver,
    int syncListenPort,
    int configServeGrpcPort,
    int telemetryListenPort)
{
    if (hasDriver && syncListenPort > 0)
    {
        // MapZbLocalDbSync returns IEndpointRouteBuilder, NOT an IEndpointConventionBuilder, so
        // .AllowAnonymous() cannot be chained onto it. An empty-prefix group carries the convention
        // to whatever it maps while leaving the gRPC route patterns byte-identical.
        app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync();
    }

    if (hasAdmin && configServeGrpcPort > 0)
    {
        app.MapGrpcService<DeploymentArtifactGrpcService>().AllowAnonymous();
        app.MapGrpcService<CentralCommandGrpcService>().AllowAnonymous();
    }

    if (hasDriver && telemetryListenPort > 0)
    {
        app.MapGrpcService<TelemetryStreamGrpcService>().AllowAnonymous();
        app.MapGrpcService<NodeCommandGrpcService>().AllowAnonymous();
    }
}

Program.cs:743-784 collapses to one call (app.MapOtOpcUaMeshGrpcServices(hasAdmin, hasDriver, syncListenPort, configServeGrpcPort, telemetryListenPort);) with the existing per-map rationale comments moved verbatim onto the new method's members — no comment is deleted; they are the record of why each endpoint rides which listener.

Alternatives considered:

RequireHost / port-scoping the mesh endpoints — does not fix this. RequireHost constrains endpoint matching, not authorization; a matched endpoint is still subject to the fallback policy. It is a plausible complement (it would stop a mesh service being reachable over the AdminUI's Traefik-fronted port), but the ports are already dedicated and HTTP/2-only, so it buys a second place to keep in sync for no authorization benefit. Rejected as the fix; not adopted as hardening.
A named authorization policy / a real PSK authentication scheme — architecturally the "correct" answer (auth in middleware, where the framework expects it) and worth naming as the eventual consolidation. Rejected here: it would duplicate five per-service key resolutions (three different option sources plus IMeshPskProvider's per-cluster map) into a scheme, and it would change the failure surface every dialer already handles from the interceptors' PermissionDenied/Unauthenticated gRPC statuses to a middleware 403 — i.e. it converts a Critical one-liner into a cross-cutting redesign of the mesh's auth contract. Recorded as a follow-up note, not this plan.
Host the mesh services on a second WebApplication with its own pipeline — true isolation, and the only option that makes the fallback structurally unable to reach them. Rejected: Program.cs:604-623 documents how delicate the current single-host Kestrel takeover already is (any explicit Listen* discards ASPNETCORE_URLS; the surface must be re-bound exactly once), and a second host duplicates the DI graph the services resolve from.
Branch UseAuthorization so it does not run for mesh paths (UseWhen) — path-sniffing middleware, evaluated before the endpoint is known in the general case, and it would silently exempt anything sharing a prefix. Rejected.
Change the FallbackPolicy to something narrower — rejected outright: the fallback is what protects every AdminUI page and hub; weakening it to fix five endpoints trades a deploy-plane outage for an authorization hole.
Extend the empty-prefix group over all five maps — tidier, but it puts four already-chainable maps behind an untested-in-this-repo route-group/gRPC interaction for no gain. Rejected; the group is scoped to the one map that needs it.

Guards (the actual deliverable):

MeshGrpcAuthStackTests (new, Host.IntegrationTests) — the CI-shaped variant of MeshGrpcBoundaryTests the action item asks for. It builds a WebApplication whose auth stack is the production registration — builder.Services.AddOtOpcUaAuth(config) with Security:Auth:DisableLogin=false, then app.UseAuthentication(); app.UseAuthorization(); — and maps the mesh services through MapOtOpcUaMeshGrpcServices. Composition constraints found while verifying (all cheap, all must be in the fixture or it will not build):
- AddOtOpcUaAuth calls AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>() (:73-75) ⇒ the fixture registers an InMemory OtOpcUaConfigDbContext (the same InMemoryConfigDbFactory idiom MeshGrpcBoundaryTests.cs:288-298 already uses, promoted to a shared fixture helper).
- Set builder.Environment.EnvironmentName = "Production" so ValidateOnBuild/ValidateScopes do not reject services (ILdapAuthService's graph) that are never resolved without a login attempt.
- Security:Auth:DisableLogin must be explicitly false — the whole point is to exercise the cookie handler, not AutoLoginAuthenticationHandler. The fixture asserts the resolved scheme handler type, so a future default flip cannot silently re-mask the test.
MeshGrpcAnonymousMetadataTests (new, same project) — an endpoint-metadata census over ((IEndpointRouteBuilder)app).DataSources after MapOtOpcUaMeshGrpcServices, in all three role shapes (admin-only, driver-only, fused). Asserts every mesh endpoint carries IAllowAnonymous, and (guard-of-the-guard) that the census found the expected endpoint count per shape — a census over zero endpoints passes vacuously, which is the exact failure class this plan exists to close. It also asserts the census is scoped: only the five known gRPC service prefixes are marked anonymous, so a future change that anonymises the AdminUI wholesale fails here.
MeshGrpcMapConventionTests (new, source scan over Program.cs, following FlushLoggingConventionTests.cs:28-45's solution-root-walk idiom) — every MapGrpcService< / MapZbLocalDbSync( occurrence in Program.cs is inside MeshGrpcEndpoints' call, i.e. Program.cs contains zero direct map sites; the scan asserts the count is 0 and that MapOtOpcUaMeshGrpcServices is called exactly once. This is the guard the metadata census cannot be: it catches a map added to Program.cs that bypasses the helper entirely.

Rig (theme 2): docker-dev/docker-compose.yml central-1 (:385) and central-2 (:511) become Security__Auth__DisableLogin: "${OTOPCUA_DISABLE_LOGIN:-true}", with a header-comment paragraph naming the production-faithful gate mode and the LDAP users it needs. Default behaviour is byte-identical.

Implementation steps

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs — new mapper (above), carrying the five map sites' existing rationale comments plus the S14 rationale.
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs — replace :743-784 with the single MapOtOpcUaMeshGrpcServices(...) call; keep the syncListenPort/configServeGrpcPort/telemetryListenPort locals and the Kestrel block (:604-711) untouched.
docker-dev/docker-compose.yml — parameterise Security__Auth__DisableLogin on central-1/central-2; document the flip in the file header.
docs/security.md — a short "Mesh gRPC ingress is AllowAnonymous by design" section: what the fallback does, why the interceptors are the gate, what fails when the opt-out is missing, and the two standing guards.
docs/Configuration.md §MeshTransport / §ConfigSource-ConfigServe and docs/Telemetry.md — one cross-reference line each, so an operator reading about the listener ports meets the constraint.

Tests

MeshGrpcAuthStackTests (new): real auth stack + MapOtOpcUaMeshGrpcServices, on an h2c Kestrel port.
- Positive (load-bearing): a DeploymentArtifactService fetch carrying the correct ConfigServe:ApiKey bearer succeeds through UseAuthentication/UseAuthorization; likewise a CentralCommandService.ReportApply and a NodeCommandService.Execute with their keys. Before the fix these fail; after it they pass. This is the only assertion that distinguishes the two builds.
- Negative control (anti-vacuity): the same fixture maps one extra gRPC service without AllowAnonymous; a correctly-keyed call to it must be refused by the cookie challenge. If this control passes trivially, the fixture's auth stack is not live and every other assertion in the class is void.
- Handler-identity guard: assert the cookie scheme resolves to the real CookieAuthenticationHandler, not AutoLoginAuthenticationHandler.
MeshGrpcAuthStackTests (status fidelity, T5): a no-PSK call under the real auth stack fails with the interceptor's detail string ("Config-serve authentication failed." / "Telemetry stream authentication failed." / "Node command authentication failed."), never a cookie challenge. ⚠️ Assert the detail, not only the status: ConfigServeAuthInterceptor uses Unauthenticated and an HTTP 401 maps to Unauthenticated too, so a status-only assertion passes on a broken build. Record the observed cookie-challenge signature (302 vs 401, and the resulting client-side status) in the plan's execution notes — it is the string the live gate greps for.
MeshGrpcAnonymousMetadataTests (new): census as designed above — per-role endpoint sets, IAllowAnonymous on each, expected counts (admin-only: 2; driver-only: 2 or 3 with a sync port; fused with all three ports: 5), and the scope assertion.
MeshGrpcDriverOnlyPipelineTests (new, T6): a driver-only shape (no AddOtOpcUaAuth, no UseAuthentication/UseAuthorization) serves NodeCommandService and TelemetryStreamService identically — pins the report's "site nodes unaffected" finding so that adding auth services to driver nodes fails a test instead of a fleet.
MeshGrpcMapConventionTests (new, T7): source scan of Program.cs.
Regression: MeshGrpcBoundaryTests, ConfigServeListenerTests, LocalDbSyncListenerTests, TelemetryListenerTests all stay green untouched (the mapper is behaviour-preserving; the boundary test keeps its own hand-built node pipeline, which is now honestly scoped to "the node side, without auth"). Full project: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.

Live gate — docker-dev rig, and it MUST run the falsification control first.

The rig runs locally (OrbStack on the Mac); AdminUI http://localhost:9200 round-robins central-1/central-2 via Traefik, so both centrals must be rebuilt/recreated together. LDAP is the shared GLAuth at 10.100.0.35:3893 (users alice/bob, password password; multi-role for the OPC UA data plane).

Step 0 — falsification control (pre-fix image; this is what makes the gate non-vacuous):

# On master (WITHOUT the fix), flip the central pair to production-faithful auth:
OTOPCUA_DISABLE_LOGIN=false docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2
open http://localhost:9200            # EXPECT a login form — if it auto-authenticates, the flip did not take
curl -s -X POST http://localhost:9200/api/deployments \
     -H "X-Api-Key: docker-dev-deploy-key" -H "Content-Type: application/json" \
     -d '{"CreatedBy":"r3-01-control"}'          # 202 Accepted (the deploy API is AllowAnonymous by design)
docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "fetch|artifact|401|302|/login|Unauthenticated" | tail
docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "ReportApply|ack|401|302" | tail
docker exec otopcua-dev-sql-1 /opt/mssql-tools18/bin/sqlcmd -C -S localhost -U sa -P 'OtOpcUa!Dev123' \
  -d OtOpcUa -I -Q "SELECT TOP 3 Id, Status, CreatedUtc FROM dbo.Deployment ORDER BY CreatedUtc DESC"

Required outcome: FAILURE. Deployment Status = 4 (TimedOut; 0 Dispatching / 1 AwaitingApplyAcks / 2 Sealed / 3 PartiallyFailed / 4 TimedOut), site-node logs showing the artifact fetch rejected at the HTTP layer, no NodeDeploymentState rows for this revision. A green deploy here means the flip did not take effect and the whole gate proves nothing. (If mssql-tools18 is absent in the image, use /opt/mssql-tools/bin/sqlcmd without -C; -I is mandatory for this schema or every statement fails Msg 1934.)

Step 1 — the fix:

docker compose -f docker-dev/docker-compose.yml build central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2
OTOPCUA_DISABLE_LOGIN=false docker compose -f docker-dev/docker-compose.yml up -d

Sign in at http://localhost:9200 as alice/password (proves the cookie stack is genuinely live), then repeat the headless deploy above. Required outcome:

Deployment reaches Status = 2 (Sealed) — not 4.
SELECT NodeId, RevisionHash, State FROM dbo.NodeDeploymentState shows all six nodes on the new revision.
Site-node logs show a successful artifact fetch (SHA-256 verified) and an apply; no /login, 302 or 401 on any mesh path.
/hosts shows fresh driver health for all six nodes — including central-1 and central-2. ⚠️ Do not accept the /alerts live pill as the telemetry evidence: it is aggregate ("live when ≥1 node stream is up"), so it stays green while precisely the two fused-node streams are broken. The per-node rows are the discriminating observation.
A driver command from the AdminUI (/hosts → Restart driver) reaches a central-pair node — the central→own-pair NodeCommandService dial that step 0 broke.

Step 2 — restore: docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2 (no env var ⇒ DisableLogin back to true); confirm :9200 auto-authenticates again. Record everything in archreview/plans/STATUS.md.

Effort + Risk

Effort: S code (one new file, one Program.cs edit, one compose line) + M verification (three new test classes; the rig session is the long pole, and it is run twice — once to fail, once to pass). Risk: low for the fix, medium for the refactor. The behavioural change is strictly additive: five endpoints stop being subject to a policy that was never intended to gate them, and nothing else on the host loses protection (the census's scope assertion pins that). The real risks are (a) the map extraction touching the boot path — mitigated by landing it as a behaviour-preserving commit with the listener/boundary suites green before any convention is added, and (b) the empty-prefix route group changing gRPC routing for LocalDbSync — mitigated by the census (metadata) plus a routing assertion in T3, with the fallback position being a cross-repo ZB.MOM.WW.LocalDb.Replication change to mark its own endpoints anonymous. The one thing that must not be skipped is the step-0 falsification control: a fixed build and an unflipped rig are indistinguishable at step 1.

Task breakdown

Branch: one branch fix/archreview-r3-01-mesh-grpc-allowanonymous off master. Ordering is strictly TDD: the refactor lands first with its scaffold green, then each guard is written to FAIL against the un-fixed convention and made to pass. T4–T7 are independent of each other once T3 lands.

Verification bar: every task has a deterministic guard; T4/T5 additionally carry falsifiability controls; T10 is the merge-blocking live gate and is only meaningful if its step-0 control fails first.

T1 — Extract `MapOtOpcUaMeshGrpcServices` (behaviour-preserving) + census scaffold

Classification: MEDIUM-RISK (boot-path refactor; no behaviour change) Estimated implement time: 10 min Parallelizable with: T8, T9; nothing else (every later task builds on the mapper) Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs (new), src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshGrpcAnonymousMetadataTests.cs (new)

Write the scaffold census: build a WebApplication per role shape (admin-only / driver-only / fused-with-all-three-ports), call MapOtOpcUaMeshGrpcServices, enumerate ((IEndpointRouteBuilder)app).DataSources.SelectMany(d => d.Endpoints), and assert the expected endpoint counts and service prefixes per shape (no metadata assertion yet). Run dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~MeshGrpcAnonymousMetadataTests" → FAIL (method does not exist).
Implement MeshGrpcEndpoints as a pure move of Program.cs:743-784 — same gates, same order, every existing comment carried over. Replace the block in Program.cs with the single call.
Filter → PASS. Then the regression net: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~MeshGrpcBoundaryTests|FullyQualifiedName~ConfigServeListenerTests|FullyQualifiedName~LocalDbSyncListenerTests|FullyQualifiedName~TelemetryListenerTests" → PASS unchanged.
Commit: refactor(host): extract MapOtOpcUaMeshGrpcServices (one authority for the five mesh maps).

T2 — Census asserts `IAllowAnonymous`; `.AllowAnonymous()` on the four gRPC maps

Classification: HIGH-RISK (the Critical's fix; changes endpoint authorization metadata) Estimated implement time: 5 min Parallelizable with: T8, T9; after T1 Files: MeshGrpcEndpoints.cs, MeshGrpcAnonymousMetadataTests.cs

Extend the census: every mesh endpoint must carry IAllowAnonymous metadata; add the guard-of-the-guard (the census must have found ≥1 endpoint in each shape, with the exact expected count) and the scope assertion (only the five known service prefixes are anonymised). Filter → FAIL for all five endpoints.
Implement .AllowAnonymous() on DeploymentArtifactGrpcService, CentralCommandGrpcService, TelemetryStreamGrpcService, NodeCommandGrpcService, with the S14 rationale in the method's XML doc.
Filter → four PASS, LocalDbSync still FAIL (expected — T3 owns it). Commit: fix(host): mesh gRPC endpoints opt out of the AdminUI cookie FallbackPolicy (archreview 03/S14).

T3 — LocalDbSync anonymous group (the non-chainable map)

Classification: medium-risk (route-group/gRPC interaction on a mapped endpoint) Estimated implement time: 5 min Parallelizable with: T8, T9; after T2 Files: MeshGrpcEndpoints.cs, MeshGrpcAnonymousMetadataTests.cs

Add the assertion that the localdb_sync.v1.LocalDbSync endpoints carry IAllowAnonymous and that their route patterns are unchanged by the group (assert the raw pattern text still begins /localdb_sync.v1.LocalDbSync/). Filter → FAIL.
Implement app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync(), with the comment recording why (the library's map returns IEndpointRouteBuilder, verified against 0.2.1 metadata, and marks nothing anonymous itself).
Filter → PASS; LocalDbSyncListenerTests → PASS unchanged. Commit: fix(host): LocalDb sync endpoint opts out of the cookie fallback via an empty-prefix anon group.

T4 — `MeshGrpcAuthStackTests`: a PSK call succeeds through the REAL auth stack

Classification: HIGH-RISK verification (this is the guard that would have caught S14) Estimated implement time: ~15 min (fixture plumbing) — the deliberately-larger task Parallelizable with: T6, T7, T8, T9; after T3 Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshGrpcAuthStackTests.cs (new)

Fixture: WebApplication with EnvironmentName = "Production", an InMemory OtOpcUaConfigDbContext (DataProtection's PersistKeysToDbContext dependency), AddOtOpcUaAuth(config) with Security:Auth:DisableLogin=false, AddGrpc with the production interceptors, UseAuthentication()/UseAuthorization(), MapOtOpcUaMeshGrpcServices, h2c Kestrel on a free port. Assert the resolved cookie-scheme handler is CookieAuthenticationHandler (not AutoLoginAuthenticationHandler).
Positive tests (run against HEAD before T2 to see them FAIL, then against the fixed build): correctly-keyed DeploymentArtifactService fetch, CentralCommandService.ReportApply, and NodeCommandService.Execute each succeed.
Negative control: one extra gRPC service mapped without AllowAnonymous in the same fixture — a correctly-keyed call to it is refused by the cookie challenge. (If it succeeds, the fixture's auth stack is inert and the class proves nothing.)
Filter → PASS. Commit: test(host): mesh gRPC boundary under the production auth stack (FallbackPolicy live).

Classification: high-risk verification (the vacuity trap) Estimated implement time: 5 min Parallelizable with: T6, T7, T8, T9; after T4 Files: MeshGrpcAuthStackTests.cs

Add: a no-bearer call to each of the three services fails with the interceptor's detail string — "Config-serve authentication failed.", "Telemetry stream authentication failed.", "Node command authentication failed." — and a wrong-bearer call likewise. ⚠️ Assert the detail, not the status alone: ConfigServeAuthInterceptor throws Unauthenticated and Grpc.Net.Client maps HTTP 401 to Unauthenticated, so a status-only assertion is green on a broken build.
Record the observed cookie-challenge signature (302 to /login vs 401, and the client-side status/detail it produces) in a code comment and in this plan's execution notes — the live gate greps for it.
Filter → PASS. Commit: test(host): no-PSK mesh calls report the interceptor's verdict, not a cookie 401/redirect.

T6 — Driver-only pipeline control (pins "site nodes unaffected")

Classification: standard (test-only) Estimated implement time: 5 min Parallelizable with: T4, T5, T7, T8, T9; after T3 Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshGrpcDriverOnlyPipelineTests.cs (new)

Build the driver-only shape (no AddOtOpcUaAuth, no UseAuthentication/UseAuthorization) and assert NodeCommandService/TelemetryStreamService behave identically to T4's fused shape — correctly-keyed call succeeds, no-key call gets the interceptor's status. Also assert the app has no IAuthorizationPolicyProvider fallback registered, which is why it is unaffected — a future "register auth everywhere" change then fails here rather than in a site outage.
Filter → PASS. Commit: test(host): driver-only nodes carry no authorization middleware (03/S14 exemption pinned).

T7 — `Program.cs` source-scan convention guard

Classification: standard (test-only) Estimated implement time: 5 min Parallelizable with: T4, T5, T6, T8, T9; after T1 Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshGrpcMapConventionTests.cs (new)

Following tests/Drivers/Cli/ZB.MOM.WW.OtOpcUa.Driver.S7.Cli.Tests/FlushLoggingConventionTests.cs:28-45 (solution-root walk + File.ReadAllText): assert Program.cs contains zero MapGrpcService< and zero MapZbLocalDbSync( occurrences, and exactly one MapOtOpcUaMeshGrpcServices( call. Guard-of-the-guard: the located file must be non-empty and must contain MapOtOpcUaMeshGrpcServices, so a bad path cannot pass the test by finding nothing.
Filter → PASS (it would have FAILED at HEAD before T1 — note that in the test's remarks). Commit: test(host): a mesh gRPC map added outside MeshGrpcEndpoints fails the build.

T8 — docker-dev: make the production-faithful auth mode a one-variable flip

Classification: standard (rig config; default behaviour unchanged) Estimated implement time: 5 min Parallelizable with: everything Files: docker-dev/docker-compose.yml

Security__Auth__DisableLogin: "${OTOPCUA_DISABLE_LOGIN:-true}" on central-1 (:385) and central-2 (:511); header-comment paragraph naming the flip, the LDAP users (alice/bob, password password, shared GLAuth 10.100.0.35:3893), and why it exists (theme 2: every mesh live gate before R3-01 ran with the blocking middleware neutralized).
Verify docker compose -f docker-dev/docker-compose.yml config resolves to "true" with no env var set (byte-identical default) and to "false" with OTOPCUA_DISABLE_LOGIN=false. Commit: chore(docker-dev): parameterise Security__Auth__DisableLogin on the central pair.

T9 — Docs

Classification: standard (docs) Estimated implement time: 5 min Parallelizable with: everything Files: docs/security.md, docs/Configuration.md, docs/Telemetry.md

Add "Mesh gRPC ingress is AllowAnonymous by design" to docs/security.md: what the cookie FallbackPolicy covers, why interceptors cannot gate what the middleware already rejected, the fail-closed property that makes anonymous safe, the affected-map table (incl. the fused-vs-driver-only distinction), and the two standing guards. One cross-reference line each in docs/Configuration.md §MeshTransport / §ConfigSource-ConfigServe and docs/Telemetry.md. Commit: docs: mesh gRPC ingress is AllowAnonymous; the PSK interceptors are the gate (R3-01).

T10 — LIVE GATE: docker-dev with `DisableLogin=false` — operator/rig task

Classification: HIGH-RISK verification gate (merge-blocking) Estimated implement time: ~45 min rig session (two passes) Parallelizable with: nothing (needs the built branch and a master build for the control) Files: none (rig session; outcomes recorded in archreview/plans/STATUS.md)

Step 0 — falsification control on the PRE-FIX build, per Finding 1's recipe: OTOPCUA_DISABLE_LOGIN=false on the central pair, headless deploy, expect Deployment.Status = 4 (TimedOut) with the fetch/ack rejections in the node logs. A green deploy here voids the entire gate — stop and find out why the flip did not take.
Step 1 — the fixed build: rebuild all six nodes, sign in as alice, headless deploy ⇒ Status = 2 (Sealed), NodeDeploymentState on the new revision for all six nodes, no /login/302/401 on any mesh path, /hosts fresh for central-1 and central-2 (not the aggregate /alerts pill — it cannot see the fused-node streams failing), and an AdminUI driver restart reaching a central-pair node.
Step 2 — restore DisableLogin to true (no env var) and confirm :9200 auto-authenticates.
Record both passes — including the exact log signature the blocked calls produced — in STATUS.md and in this plan's execution notes.

Sequencing summary

T1 → T2 → T3                 (fix, sequential — same two files)
T3 → T4 → T5                 (real-auth-stack guard + status fidelity)
T3 → T6                      (driver-only control)
T1 → T7                      (source scan)
T8, T9 anytime
T10 last — and its step-0 control runs on master BEFORE the branch is merged

Overall effort: ~half a dev-day code+tests, plus the two-pass rig session. The Critical itself is five lines; everything else in this plan exists because the five lines were invisible to a seven-phase live-gate program.

Execution notes (T1–T9, 2026-08-20)

Branch fix/r3-01-mesh-grpc-allowanonymous, one commit per task as specified. T1–T9 all landed; T10 (live gate) left pending for the orchestrator per the run brief. Every code-touching task (T1–T7) was verified red-before-fix by hand — for T2–T5 by asserting against the un-fixed MeshGrpcEndpoints.cs/Program.cs (via a temporary git show <parent-commit>:<path> swap, rebuild, run, then restore — confirmed byte-identical restore via git diff --stat each time) and observing the expected failure before reapplying the fix. Two things surfaced during that process that are worth recording:

The census's endpoint filter could not stay a "route pattern starts with /" heuristic once T3's empty-prefix MapGroup landed. RealMethodEndpoints in MeshGrpcAnonymousMetadataTests originally filtered on RoutePattern.RawText?.StartsWith('/') to exclude grpc-dotnet's own per-app "Unimplemented service" catch-all and per-service "Unimplemented method for X" fallback (neither carries a route pattern starting with / in the ungrouped case). Once LocalDbSync moved behind app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync(), those SAME fallback patterns also gained a leading / inside the group, so the heuristic started over-counting (DriverOnly_WithASyncPort_AlsoMapsTheLocalDbSyncEndpoint and LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical both went red for the wrong reason — 3 endpoints found where 1 was expected). Fixed by matching against the exact known set of five method route-pattern literals instead of a shape heuristic. Not a plan defect — the plan didn't specify the filter mechanism — but a genuine gotcha for anyone extending this census: grpc-dotnet's route-pattern-shape for its own fallback endpoints is not stable across a plain MapGrpcService vs. a grouped one, so don't filter on shape once any mesh map moves behind a group.
T4's fixture does not map or wire LocalDbSyncService's real dependencies. The plan's fixture description says "maps the mesh services through MapOtOpcUaMeshGrpcServices"; T4's MeshGrpcAuthStackTests calls it with syncListenPort: 0 (LocalDbSync NOT mapped by the helper) and instead hand-maps the library's LocalDbSyncService directly, with no .AllowAnonymous(), purely as the negative control. This is a minimal, deliberate narrowing: no assertion in T4 or T5 exercises LocalDbSync's real RPC (the plan's own positive/negative lists never name it), and standing up its DI graph (AddZbLocalDb/AddZbLocalDbReplication, a temp-file SQLite ILocalDb) would have added fixture weight with nothing to verify — the authorization middleware refuses an unauthorized call before the service is ever constructed, so the DI graph doesn't need to exist for the negative control to be meaningful. Recorded here because it's a case where the fixture is deliberately narrower than a literal reading of the plan's prose, even though every named assertion is still covered.
The positive/negative service lists T4 and T5 assert are asymmetric, and that asymmetry is the plan's own, not introduced here. T4's positive list is {DeploymentArtifactService.Fetch, CentralCommandService.ReportApply, NodeCommandService.Execute} — Telemetry is not asserted positively there. T5's no-PSK detail-string list is {Config-serve, Telemetry stream, Node command} — CentralCommandService's own detail string ("Central command authentication failed.") is never asserted anywhere in T4/T5. Implemented exactly as specified (no scope widening per the run brief); flagging the gap in case a follow-up wants a NoOrWrongPsk_CentralCommandReportApply_FailsWithTheInterceptorsDetail case for full symmetry — trivial to add given the fixture already maps and keys that service.
Observed cookie-challenge signature (T5 step 2), captured directly against this repo's fixture stack (macOS, Kestrel h2c, Grpc.Net.Client over HttpClient/SocketsHttpHandler): every blocked positive-test call in the red state failed as RpcException(StatusCode.Internal, "Error starting gRPC call. HttpRequestException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)") — never Unauthenticated/PermissionDenied. The cookie handler's redirect/401 breaks the prior-knowledge h2c stream before any gRPC trailers are written, so the client sees a transport-level reset rather than a clean gRPC status. The live gate (T10) should grep node/central logs and browser network traces for PROTOCOL_ERROR, /login, 302 and 401 together — not for 401 alone, and not for a gRPC Unauthenticated/ PermissionDenied status, which this environment never actually produces for the blocked case. (A real reverse-proxy in front of a production deployment could still surface a clean HTTP 401 instead of a stream reset, depending on how it terminates HTTP/2 — the plan's own multi-signal grep advice already covers that; this note is the empirical confirmation of one concrete shape.)
T6 passes identically on both the pre-fix and post-fix mapper (confirmed by the same temporary-swap procedure). Driver-only nodes never register AddOtOpcUaAuth at all, so .AllowAnonymous() presence or absence on the mesh maps has zero effect on that shape — T6 is a structural pin/regression-control test, not a red-then-green TDD test for a behaviour T1–T3 changed. This matches the plan's own "standard (test-only)" classification; recorded because the run brief asked for red-before-fix confirmation to be checked (and reported) for every task, and this is the one task where "red" never existed to begin with.
T8's docker compose config verification ran locally on macOS (OrbStack-backed Docker Compose v5.1.2) — no Windows-VM lmxopcua-fix helper needed, consistent with the Mac-path guidance in CLAUDE.md. Confirmed both branches: unset OTOPCUA_DISABLE_LOGIN resolves Security__Auth__DisableLogin: "true" on both central-1/central-2; OTOPCUA_DISABLE_LOGIN=false resolves "false" on both.

No other deviations from the plan's design or task breakdown. T10 was not run — left pending per the run brief, for the orchestrator.

Live gate result (2026-08-20) — PASS on docker-dev with `DisableLogin=false` (control failed first)

Rig: docker-dev/docker-compose.yml on OrbStack (Mac), fresh volumes (down -v first), six host containers. Both runs used OTOPCUA_DISABLE_LOGIN=false on the central pair (verified in-container: Security__Auth__DisableLogin=false on central-1/central-2; curl -s http://localhost:9200/ → 302 → /login?ReturnUrl=%2F in both runs — the cookie FallbackPolicy was live for the UI throughout). Deploys were triggered with ./docker-dev/deploy.sh (POST /api/deployments, X-Api-Key: docker-dev-deploy-key), which answered 202 Accepted in both runs — the headless deploy API is AllowAnonymous + key-gated and is NOT behind the cookie fallback, so the API-key path works with login enabled (one open question from the brief, answered: not a finding).

Step 0 — falsification control: FAILED as required

Control image otopcua-host:prefix = 53c41fc6bc01, built from a scratch worktree of master f1b976f5 with the five R3-01 opt-outs reverted in MeshGrpcEndpoints.cs (.AllowAnonymous() removed from the four MapGrpcService maps, the empty-prefix MapGroup(...).AllowAnonymous() collapsed back to a bare MapZbLocalDbSync(); git diff --stat = 1 file, 5 insertions/5 deletions). Chosen over a checkout of 851e4cf5 so the control image matched the R3-04 compose (site-b-2 Cluster:Roles-only, site-a-2 miscased pin) — the same compose both runs used. Tagged over otopcua-host:dev for the run; image parity asserted on all six containers (docker inspect --format '{{.Image}}' = 53c41fc6bc01… ×6).

Deployment 26077962… (revision 9e3540a1…): Status = 4 (TimedOut), SealedAtUtc = NULL, six NodeDeploymentState rows all Status = 0, AppliedAtUtc = NULL.
central-1 (seal): [16:10:04 WRN] Deployment 26077962e53a4ae99538c5aaaa396a85 timed out after 00:02:00 (0/6 acks landed). No ack from: central-1:4053, central-2:4053, site-a-1:4053, site-a-2:4053, site-b-1:4053, site-b-2:4053.
central-1 (own-pair command dial, the fused-node path): [16:08:04 WRN] Mesh: DispatchDeployment (correlationId=9fe30594…) was NOT delivered to central-1:4053 (Cancelled: Bad gRPC response. Invalid content-type value: text/html; charset=utf-8). — and identically for central-2:4053.
site-a-1 (artifact fetch + ack): [16:08:04 WRN] Artifact fetch for 26077962… from http://central-1:4055 failed (Cancelled); trying next endpoint. → … failed on ALL 2 endpoint(s); returning no bytes (apply failure — last-known-good is kept). → [16:08:04 ERR] DriverHost site-a-1:4053: FetchAndCache apply of 26077962… FAILED … nothing was applied. Staying on revision null → [16:08:04 WRN] Dropping the ApplyAck for 26077962… from site-a-1:4053: all 4 central endpoint(s) refused it (http://central-1:4055 => Cancelled; …).

Server-side signature, per central (Kestrel request log, sort | uniq -c over the whole control run):

endpoint (central-1 / central-2)	responded
`POST /deployment_artifact.v1.DeploymentArtifactService/Fetch`	302 ×4 / ×4
`POST /mesh_transport.v1.CentralCommandService/ReportApply`	302 ×8 / ×8
`POST /mesh_transport.v1.NodeCommandService/Execute`	302 ×1 / ×1
`POST /telemetry.v1.TelemetryStreamService/Subscribe`	302 ×107 / ×105 (central's own-pair dial reconnecting every ~5 s)

Site nodes (no auth stack) served NodeCommandService/Execute → 200 — the report's "site nodes unaffected" finding holds live; it is the central side of every mesh call that was gated.

Step 1 — the fix: PASS

Image otopcua-host:dev = b344565cad3d, built 12:28 local via docker compose build from the master working tree. ⚠️ Master advanced from f1b976f5 to e280db37 (R3-02 merge + one follow-up) while the control was running, and the build ran after that, so the post-fix image is master e280db37 — which contains d79d51dd (R3-01) unchanged; the working tree was clean at build time. Parity asserted on all six containers (b344565cad3d ×6), DisableLogin=false still on both centrals, SQL volume kept (so the TimedOut revision re-dispatched as a new deployment).

Deployment 98ea226a… (same revision 9e3540a1…): Status = 2 (Sealed), CreatedAtUtc 16:30:43.743, SealedAtUtc 16:30:43.920 (177 ms). Six NodeDeploymentState rows, all Status = 1 with AppliedAtUtc 16:30:43.836–.910: central-1, central-2, site-a-1, site-a-2, site-b-1, site-b-2. A second deploy after recreating the centrals (49bcf361…) sealed identically.
site-a-1: [16:30:43 INF] Fetched + verified artifact for 98ea226a… from http://central-1:4055 (1469 bytes). → cached applied deployment 98ea226a… as this node's local fallback.
Centrals' request log, same grep: DeploymentArtifactService/Fetch 200 ×4, CentralCommandService/ReportApply 200 ×8, NodeCommandService/Execute 200. Count of responded (302|401)|PROTOCOL_ERROR|/login|Invalid content-type across all six node logs after the recreate: 0 on every node except one HTTP GET / responded 302 on central-2 — the curl probe of the AdminUI root, i.e. the FallbackPolicy still active for the UI (the intended scope). The four site nodes: 0.
Signed in through the real form at http://localhost:9200/login as multi-role/password (see the LDAP finding below for why not alice); session badge listed all roles. /hosts showed the MAIN cluster's pymodbus (MAIN-modbus) driver Healthy, Last read 16:37:23 UTC then 16:37:53 UTC across two auto-refreshes — that row is fed only by central's TelemetryDial into its own pair's TelemetryStreamService, the stream that answered 302 ×212 in the control. /alerts pill live (recorded, but per the plan not relied on). The rig seed has no drivers on SITE-A/SITE-B, so there are no per-node site rows to show; site-node telemetry is instead evidenced by central-1's dial log — one boot-time Telemetry stream to node … failed (failure streak 1); reconnecting per node at recreate (connection refused, peer not yet listening) and no further failure for the rest of the run, and by the TelemetryStreamService/Subscribe streams on the centrals never closing (0 responded lines after recreate — a stream logs only when it ends).
/hosts → Controls → Restart → Confirm restart on MAIN-modbus: central-2 [16:38:19 INF] AdminOps: RestartDriver dispatched for MAIN-modbus by multi-role; both central-1 and central-2 logged HTTP POST /mesh_transport.v1.NodeCommandService/Execute responded 200 followed by DriverHost central-N:4053: restarting driver MAIN-modbus by request of multi-role — the central→own-pair NodeCommandService dial that step 0 broke (Invalid content-type value: text/html).

LocalDb sync (site-a replicating pair, step 2 of the brief): the library logs no success line, so the evidence is (a) an ESTABLISHED TCP session on site-a-2's :9001 sync listener from site-a-1 (/proc/net/tcp6 inside site-a-2: …FFFF00000761A8C0:2329 ← …FFFF00000861A8C0:E276 state 01), and (b) exactly one Replication session to peer http://site-a-2:9001 faulted; reconnecting. on site-a-1 at boot (16:29:47, inner SocketException (111): Connection refused — peer not yet listening) and none afterwards. Site nodes carry no auth stack, so this was expected to be unaffected in both runs; it was (same shape in the control).

Step 2 — restore

docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2 with no env var → Security__Auth__DisableLogin=true in both containers, curl -s http://localhost:9200/ → 200 (auto- authenticated). Rig left up on otopcua-host:dev = b344565cad3d ×6. Scratch worktree /tmp/otopcua-prefix removed. The otopcua-host:prefix image tag is left behind for re-runs.

Findings (new, outside R3-01's fix)

The rig's LDAP login is broken by the SEC-36 rotation, masked until now by DisableLogin=true. With login enabled, every sign-in failed /login?error=Unexpected%20authentication%20error (= LdapAuthFailure.ServiceAccountBindFailed). The shared GLAuth at 10.100.0.35:3893 logged invalid credentials binddn=cn=serviceaccount,dc=zb,dc=local src=10.100.0.199 — its serviceaccount passsha256 (e9c2967c…, rotated 2026-08-07, scadaproj aada53b) no longer matches the serviceaccount123 the compose hardcodes (sha256 = af29d0e5…); the directory also has no alice (that user is the Host.IntegrationTests harness GLAuth). Same class as the ScadaBridge rig breakage recorded in the scadaproj memory. The gate proceeded with a gate-only compose override (Security__Ldap__Server: host.docker.internal, pointing at the local zb-ldap-primary pair which keeps serviceaccount123) and multi-role/password; the override was dropped at restore. The compose's Security__Ldap__* block, the docker-dev/README.md "Auth" section and the plan's alice/password recipe all need correcting — filed as a follow-up, not fixed here.
The live cookie-challenge signature differs from the unit-test one. Execution note 4 recorded StatusCode.Internal … PROTOCOL_ERROR from the in-process fixture. On the rig the server side is a plain 302 (Kestrel log), and the gRPC client side surfaces as Cancelled: Bad gRPC response. Invalid content-type value: text/html; charset=utf-8 (the DispatchDeployment warning) or a bare (Cancelled) (the fetcher's per-endpoint warning). Grep for responded 302 on the serving side and Invalid content-type value: text/html / Cancelled on the dialling side; PROTOCOL_ERROR never appeared in any container log.
The seed gives the site clusters no drivers, so /hosts cannot show per-node driver health for site nodes — plan step-1 item 4 ("all six nodes") is only satisfiable for the central pair on this rig as seeded. Not a defect; noted so the next gate does not read "No drivers." as a telemetry break.

62 KiB Raw Blame History Unescape Escape

Design + Implementation Plan — R3-01 Mesh gRPC AllowAnonymous