scadaproj/docs/plans/2026-08-10-ldap-backup-dc-failover.md

LDAP Backup-DC Failover Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Add sticky-preference multi-endpoint failover to ZB.MOM.WW.Auth.Ldap so a login survives a primary-DC outage (design: 2026-08-10-ldap-backup-dc-failover-design.md).

Architecture: One additive LdapOptions.FallbackServers list; LdapAuthService (already a singleton) walks an ordered endpoint list on system-side failures only, remembering the last endpoint that gave an authoritative answer in a volatile int. Per-attempt connections come from the existing ILdapConnectionFactory (one fresh connection per endpoint attempt — Novell connections are not safely re-connectable after failure). New optional ILogger<LdapAuthService> makes failover visible.

Tech Stack: .NET 10, Novell.Directory.Ldap.NETStandard, xunit (+ Xunit.SkippableFact for the GLAuth integration suite), central package management (Directory.Packages.props), family version in Directory.Build.props.

Repo/branch: All work in ~/Desktop/scadaproj (the Auth lib is a plain directory of this repo — do NOT git init anything). Create branch feat/auth-ldap-failover off main; merge back + push at the end (Task 8).

Working directory for all builds/tests: /Users/dohertj2/Desktop/scadaproj/ZB.MOM.WW.Auth Test command: dotnet test ZB.MOM.WW.Auth.slnx (baseline: 215 passed + 1 skipped, 0 warnings — TreatWarningsAsErrors conventions apply; verify baseline before starting).

One deliberate refinement over the design doc (record in the design doc in Task 4): sticky preference updates on ANY authoritative answer (success or user-side failure like UserNotFound/BadCredentials), not only on success — a DC that answers is healthy, and updating only on success would re-pay the primary's connect timeout on every failed-password attempt during an outage.

---

Task 1: LdapEndpoint + LdapEndpointParser

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 3

Files:

• Create: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/Internal/LdapEndpoint.cs
• Test: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapEndpointParserTests.cs (create)

Step 1: Write the failing tests

using ZB.MOM.WW.Auth.Ldap.Internal;

namespace ZB.MOM.WW.Auth.Ldap.Tests;

public class LdapEndpointParserTests
{
    [Theory]
    [InlineData("dc2.example.com", 636, "dc2.example.com", 636)]      // bare host -> default port
    [InlineData("dc2.example.com:3894", 636, "dc2.example.com", 3894)] // explicit port wins
    [InlineData("  dc2.example.com : 3894  ", 636, "dc2.example.com", 3894)] // whitespace tolerated
    public void TryParse_ValidEntry_Succeeds(string entry, int defaultPort, string host, int port)
    {
        Assert.True(LdapEndpointParser.TryParse(entry, defaultPort, out var ep, out _));
        Assert.Equal(host, ep.Host);
        Assert.Equal(port, ep.Port);
    }

    [Theory]
    [InlineData("")]              // empty
    [InlineData("   ")]           // whitespace
    [InlineData(":636")]          // empty host
    [InlineData("dc2:0")]         // port below range
    [InlineData("dc2:65536")]     // port above range
    [InlineData("dc2:abc")]       // non-numeric port
    [InlineData("2001:db8::1:636")] // IPv6 literal (unsupported, >1 colon)
    public void TryParse_InvalidEntry_Fails(string entry)
    {
        Assert.False(LdapEndpointParser.TryParse(entry, 636, out _, out var error));
        Assert.NotEmpty(error);
    }

    [Fact]
    public void Endpoint_ToString_IsHostColonPort() =>
        Assert.Equal("dc2:3894", new LdapEndpoint("dc2", 3894).ToString());
}

Step 2: Run to verify failure

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapEndpointParserTests" Expected: FAIL to compile (LdapEndpoint not defined).

Step 3: Implement

namespace ZB.MOM.WW.Auth.Ldap.Internal;

/// <summary>A single LDAP server endpoint resolved from configuration.</summary>
internal sealed record LdapEndpoint(string Host, int Port)
{
    public override string ToString() => $"{Host}:{Port}";
}

/// <summary>
/// Parses <c>FallbackServers</c> entries: <c>"host"</c> (adopts the caller's default port)
/// or <c>"host:port"</c>. IPv6 literals are not supported — use a DNS name.
/// </summary>
internal static class LdapEndpointParser
{
    public static bool TryParse(string entry, int defaultPort, out LdapEndpoint endpoint, out string error)
    {
        endpoint = null!;
        error = "";

        if (string.IsNullOrWhiteSpace(entry))
        {
            error = "entry is empty or whitespace";
            return false;
        }

        var parts = entry.Trim().Split(':');
        if (parts.Length > 2)
        {
            error = $"entry \"{entry.Trim()}\" contains more than one ':' " +
                    "(IPv6 literals are not supported — use a DNS name)";
            return false;
        }

        var host = parts[0].Trim();
        if (host.Length == 0)
        {
            error = $"entry \"{entry.Trim()}\" has an empty host";
            return false;
        }

        var port = defaultPort;
        if (parts.Length == 2 && (!int.TryParse(parts[1].Trim(), out port) || port is < 1 or > 65535))
        {
            error = $"entry \"{entry.Trim()}\" has an invalid port (must be 1–65535)";
            return false;
        }

        endpoint = new LdapEndpoint(host, port);
        return true;
    }
}

Step 4: Run tests, verify pass

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapEndpointParserTests" Expected: 11 PASS.

Step 5: Commit

git add src/ZB.MOM.WW.Auth.Ldap/Internal/LdapEndpoint.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapEndpointParserTests.cs
git commit -m "feat(ldap): LdapEndpoint + FallbackServers entry parser"

---

Task 2: LdapOptions.FallbackServers + validator rules

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 3 (NOT Task 1 — uses the parser)

Files:

• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs (add property after ConnectionTimeoutMs, line ~20)
• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs (new check after the ServiceAccountDn check)
• Test: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapOptionsValidatorTests.cs (append)

Step 1: Write the failing tests (append to LdapOptionsValidatorTests — match its terse Opts() with {} style)

    [Fact]
    public void Validator_Succeeds_WithValidFallbackServers() =>
        Assert.False(new LdapOptionsValidator()
            .Validate(null, Opts() with { FallbackServers = new[] { "dc2", "dc3:3894" } })
            .Failed);

    [Theory]
    [InlineData("")]
    [InlineData(":636")]
    [InlineData("dc2:0")]
    [InlineData("dc2:notaport")]
    public void Validator_Fails_OnMalformedFallbackServer(string entry)
    {
        var result = new LdapOptionsValidator()
            .Validate(null, Opts() with { FallbackServers = new[] { entry } });

        Assert.True(result.Failed);
        Assert.Contains(nameof(LdapOptions.FallbackServers), result.FailureMessage);
    }

    [Fact]
    public void Validator_Skips_FallbackServers_WhenDisabled() =>
        Assert.False(new LdapOptionsValidator()
            .Validate(null, Opts() with { Enabled = false, FallbackServers = new[] { ":::" } })
            .Failed);

Step 2: Run to verify failure

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapOptionsValidatorTests" Expected: FAIL to compile (FallbackServers not defined).

Step 3: Implement

In LdapContracts.cs, add to LdapOptions after ConnectionTimeoutMs:

    /// <summary>
    /// Ordered fallback LDAP endpoints tried when the primary (and any earlier fallback)
    /// fails with a system-side error (connect/TLS, service-account bind, or search).
    /// Each entry is <c>"host"</c> (adopts <see cref="Port"/>) or <c>"host:port"</c>.
    /// Empty (the default) disables failover entirely — single-endpoint behavior is
    /// unchanged. The TLS validation policy (<see cref="ServerCertificateValidationCallback"/>
    /// / <see cref="AllowInsecure"/>) applies to every endpoint. Endpoint preference is
    /// sticky: the service keeps using the last endpoint that answered until it fails.
    /// </summary>
    public IReadOnlyList<string> FallbackServers { get; init; } = Array.Empty<string>();

In LdapOptionsValidator.Validate, after the ServiceAccountDn check (before the final return):

        foreach (var entry in options.FallbackServers)
        {
            if (!Internal.LdapEndpointParser.TryParse(entry, options.Port, out _, out var error))
            {
                return ValidateOptionsResult.Fail(
                    $"{nameof(LdapOptions.FallbackServers)} contains an invalid entry — {error}. " +
                    "Each entry must be \"host\" or \"host:port\" (port 1–65535).");
            }
        }

Also extend the class <remarks> bullet list with a fifth condition (fallback entries must parse).

Step 4: Run tests, verify pass

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapOptionsValidatorTests" Expected: all PASS (existing + 6 new).

Step 5: Commit

git add src/ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs src/ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapOptionsValidatorTests.cs
git commit -m "feat(ldap): FallbackServers option + boot-time validation"

---

Task 3: SequenceLdapConnectionFactory test double

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2

The existing FakeLdapConnectionFactory returns ONE shared fake — useless for failover tests, where each endpoint attempt gets a fresh connection and the fakes must differ per attempt. Do NOT modify FakeLdapConnection (its BindAttempts == 1 service-bind heuristic is per-instance and stays valid because the walk creates a fresh connection per attempt).

Files:

• Create: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/SequenceLdapConnectionFactory.cs

Step 1: Implement (with inline smoke test, mirroring FakeLdapConnection.cs's pattern)

using ZB.MOM.WW.Auth.Ldap.Internal;

namespace ZB.MOM.WW.Auth.Ldap.Tests;

/// <summary>
/// Test double for <see cref="ILdapConnectionFactory"/> that hands out pre-scripted
/// <see cref="FakeLdapConnection"/> instances in order, one per <see cref="Create"/> call —
/// exactly matching the failover walk, which opens a fresh connection per endpoint attempt.
/// Records every created connection for inspection. When the script runs out, returns a
/// bare unscripted fake (which yields user-not-found) so a miscounted test fails loudly
/// rather than reusing a stale fake.
/// </summary>
internal sealed class SequenceLdapConnectionFactory : ILdapConnectionFactory
{
    private readonly Queue<FakeLdapConnection> _script;

    public SequenceLdapConnectionFactory(params FakeLdapConnection[] fakes)
        => _script = new Queue<FakeLdapConnection>(fakes);

    /// <summary>Every connection handed out, in creation order (scripted or overflow).</summary>
    public List<FakeLdapConnection> Created { get; } = new();

    public ILdapConnection Create()
    {
        var fake = _script.Count > 0 ? _script.Dequeue() : new FakeLdapConnection();
        Created.Add(fake);
        return fake;
    }
}

public class SequenceLdapConnectionFactorySmokeTests
{
    [Fact]
    public void Create_ReturnsScriptedFakesInOrder_ThenOverflows()
    {
        var first = new FakeLdapConnection().ThrowOnConnect();
        var second = new FakeLdapConnection();
        var factory = new SequenceLdapConnectionFactory(first, second);

        Assert.Same(first, factory.Create());
        Assert.Same(second, factory.Create());
        Assert.NotSame(second, factory.Create()); // overflow: fresh bare fake
        Assert.Equal(3, factory.Created.Count);
    }
}

Step 2: Run, verify pass

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~SequenceLdapConnectionFactorySmokeTests" Expected: 1 PASS.

Step 3: Commit

git add tests/ZB.MOM.WW.Auth.Ldap.Tests/SequenceLdapConnectionFactory.cs
git commit -m "test(ldap): sequence connection factory for failover tests"

---

Task 4: Endpoint walk + sticky preference in LdapAuthService

Classification: high-risk (auth-critical logic; the fail-closed contract must survive the restructure) Estimated implement time: ~5 min Parallelizable with: none (depends on Tasks 1–3)

Files:

• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs (restructure Authenticate; keep every stage's behavior byte-identical inside the per-endpoint attempt)
• Test: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs (create)
• Modify (doc note only): scadaproj/docs/plans/2026-08-10-ldap-backup-dc-failover-design.md — record the sticky-on-any-authoritative-answer refinement.

NOTE: the ILogger parameter arrives in Task 5 — in this task the service compiles WITHOUT logging (leave the _logger? lines out; Task 5 adds them). Existing tests in LdapAuthServiceTests / LdapAuthServiceFailureTests must pass unchanged — they use single-endpoint options and the shared-instance FakeLdapConnectionFactory, which still works because a single-endpoint walk calls Create() once.

Step 1: Write the failing tests

using ZB.MOM.WW.Auth.Abstractions.Ldap;

namespace ZB.MOM.WW.Auth.Tests; // match the existing LdapAuthServiceTests namespace — check and mirror it

public class LdapAuthServiceFailoverTests
{
    private static LdapOptions Opts(params string[] fallbacks) => new()
    {
        Enabled = true,
        Server = "dc1",
        Port = 636,
        Transport = LdapTransport.Ldaps,
        SearchBase = "dc=x",
        ServiceAccountDn = "cn=svc,dc=x",
        ServiceAccountPassword = "svcpw",
        FallbackServers = fallbacks,
    };

    private static FakeLdapConnection HealthyWithUser() =>
        new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=ops,dc=x" });

    // ---- failover triggers (system-side) ----

    [Fact]
    public async Task ConnectFailure_FailsOverToFallback_AndSucceeds()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser());
        var svc = new LdapAuthService(Opts("dc2:3894"), factory);

        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.True(result.Succeeded);
        Assert.Equal(2, factory.Created.Count);
        Assert.Equal("dc1", factory.Created[0].ConnectArgs!.Value.Host);
        Assert.Equal("dc2", factory.Created[1].ConnectArgs!.Value.Host);
        Assert.Equal(3894, factory.Created[1].ConnectArgs!.Value.Port);
    }

    [Fact]
    public async Task ServiceBindFailure_FailsOverToFallback()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnServiceBind(), HealthyWithUser());
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.True(result.Succeeded);
        Assert.Equal(636, factory.Created[1].ConnectArgs!.Value.Port); // bare host adopts Port
    }

    // ---- user-side results NEVER fail over ----

    [Fact]
    public async Task BadUserCredentials_DoesNotFailOver()
    {
        var factory = new SequenceLdapConnectionFactory(
            HealthyWithUser().ThrowOnUserBind());
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var result = await svc.AuthenticateAsync("alice", "wrong", CancellationToken.None);

        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
        Assert.Single(factory.Created);
    }

    [Fact]
    public async Task UserNotFound_DoesNotFailOver()
    {
        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection());
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var result = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None);

        Assert.Equal(LdapAuthFailure.UserNotFound, result.Failure);
        Assert.Single(factory.Created);
    }

    [Fact]
    public async Task GroupLookupFailed_DoesNotFailOver()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", Array.Empty<string>()));
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
        Assert.Single(factory.Created);
    }

    // ---- exhaustion ----

    [Fact]
    public async Task AllEndpointsDown_ReturnsServiceAccountBindFailed()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(),
            new FakeLdapConnection().ThrowOnConnect());
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
        Assert.Equal(2, factory.Created.Count);
    }

    // ---- stickiness ----

    [Fact]
    public async Task StickyPreference_SecondLoginStartsAtLastGoodEndpoint()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
            HealthyWithUser(),                          // login 1 @ dc2: answers
            HealthyWithUser());                         // login 2: should go straight to dc2
        var svc = new LdapAuthService(Opts("dc2"), factory);

        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);

        Assert.Equal(3, factory.Created.Count);
        Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host);
    }

    [Fact]
    public async Task StickyPreference_DriftsBackToPrimary_WhenPreferredFails()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
            HealthyWithUser(),                          // login 1 @ dc2: answers -> preferred
            new FakeLdapConnection().ThrowOnConnect(), // login 2 @ dc2: now dead
            HealthyWithUser(),                          // login 2 @ dc1 (declared order): recovered
            HealthyWithUser());                         // login 3: should start at dc1 again
        var svc = new LdapAuthService(Opts("dc2"), factory);

        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);

        Assert.Equal(5, factory.Created.Count);
        Assert.Equal("dc1", factory.Created[3].ConnectArgs!.Value.Host);
        Assert.Equal("dc1", factory.Created[4].ConnectArgs!.Value.Host);
    }

    [Fact]
    public async Task StickyPreference_UpdatesOnUserSideAnswer_NotOnlySuccess()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
            new FakeLdapConnection(),                   // login 1 @ dc2: answers UserNotFound
            HealthyWithUser());                         // login 2: should start at dc2
        var svc = new LdapAuthService(Opts("dc2"), factory);

        var first = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None);
        Assert.Equal(LdapAuthFailure.UserNotFound, first.Failure);

        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
        Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host);
    }

    // ---- back-compat ----

    [Fact]
    public async Task NoFallbacks_SingleAttempt_UnchangedBehavior()
    {
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect());
        var svc = new LdapAuthService(Opts(/* no fallbacks */), factory);

        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
        Assert.Single(factory.Created);
    }

    [Fact]
    public async Task MalformedFallbackEntry_IsSkippedDefensively()
    {
        // The validator rejects this at boot; a service constructed without validation
        // must degrade to the parseable subset, not throw.
        var factory = new SequenceLdapConnectionFactory(HealthyWithUser());
        var svc = new LdapAuthService(Opts(":::"), factory);

        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
        Assert.Single(factory.Created);
    }
}

Before writing: open LdapAuthServiceTests.cs, mirror its exact namespace and the internal-visibility usage (the test project has InternalsVisibleTo — LdapAuthService's internal ctor and the fakes are reachable).

Step 2: Run to verify failure

Run: dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapAuthServiceFailoverTests" Expected: FAIL — walk/stickiness not implemented (connect-failure tests fail with a single Created entry; sticky tests fail).

Step 3: Implement — restructure LdapAuthService:

1. New fields + endpoint construction:

    private readonly IReadOnlyList<LdapEndpoint> _endpoints;
    private int _preferredIndex; // index into _endpoints; volatile access; benign races accepted

In BOTH constructors' shared path (the internal one; the public one chains to it), set _endpoints = BuildEndpoints(options); and add:

    /// <summary>
    /// Primary first, then fallbacks in declared order. The validator rejects malformed
    /// entries at boot; unparseable entries are skipped defensively here so a service
    /// constructed without options validation degrades to the parseable subset.
    /// </summary>
    private static IReadOnlyList<LdapEndpoint> BuildEndpoints(LdapOptions options)
    {
        var endpoints = new List<LdapEndpoint> { new(options.Server, options.Port) };
        foreach (var entry in options.FallbackServers)
        {
            if (LdapEndpointParser.TryParse(entry, options.Port, out var ep, out _))
                endpoints.Add(ep);
        }
        return endpoints;
    }

2. Authenticate becomes the walk (steps 1–3 — gate, null-username, trim — stay as-is above it):

    private LdapAuthResult Authenticate(string username, string password)
    {
        if (!_options.Enabled)
            return LdapAuthResult.Fail(LdapAuthFailure.Disabled);
        if (string.IsNullOrWhiteSpace(username))
            return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
        username = username.Trim();

        var preferred = Volatile.Read(ref _preferredIndex);
        if ((uint)preferred >= (uint)_endpoints.Count)
            preferred = 0; // defensive: index can never exceed the immutable list, but fail safe

        var lastSystemSideFailure = LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed);
        foreach (var index in WalkOrder(_endpoints.Count, preferred))
        {
            var result = AuthenticateAgainst(_endpoints[index], username, password);

            // ServiceAccountBindFailed is the (sole) system-side bucket: the endpoint gave no
            // authoritative answer — try the next one. Every other outcome (success, or a
            // user-side failure like UserNotFound/BadCredentials) came from a healthy
            // directory and is final.
            if (result.Failure == LdapAuthFailure.ServiceAccountBindFailed)
            {
                lastSystemSideFailure = result;
                continue;
            }

            // Sticky preference: any authoritative answer proves the endpoint healthy —
            // updating only on Succeeded would re-pay the dead primary's connect timeout
            // on every mistyped password during an outage.
            if (index != preferred)
                Volatile.Write(ref _preferredIndex, index);

            return result;
        }

        return lastSystemSideFailure;
    }

    /// <summary>Preferred endpoint first, then the rest in declared order.</summary>
    private static IEnumerable<int> WalkOrder(int count, int preferred)
    {
        yield return preferred;
        for (var i = 0; i < count; i++)
        {
            if (i != preferred)
                yield return i;
        }
    }

3. AuthenticateAgainst(LdapEndpoint endpoint, ...) is the CURRENT stages 4–10 verbatim, with three mechanical changes: (a) conn.Connect(endpoint.Host, endpoint.Port, ...) instead of _options.Server/_options.Port; (b) the user-bind and group-extraction failures return directly instead of throwing StageFailure (equivalent, simpler); (c) StageFailure gains a string Stage member ("connect" / "service-bind" / "search") consumed by Task 5's logging — carry it now so Task 5 is additive:

    private LdapAuthResult AuthenticateAgainst(LdapEndpoint endpoint, string username, string password)
    {
        try
        {
            using var conn = _connectionFactory.Create();

            try
            {
                conn.Connect(endpoint.Host, endpoint.Port, _options.Transport,
                    _options.AllowInsecure, _options.ConnectionTimeoutMs,
                    _options.ServerCertificateValidationCallback);
            }
            catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "connect"); }

            try { conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword); }
            catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "service-bind"); }

            IReadOnlyList<LdapSearchEntry> entries;
            try
            {
                var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})";
                entries = conn.Search(_options.SearchBase, filter, BuildSearchAttributes());
            }
            catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "search"); }

            if (entries.Count == 0) return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound);
            if (entries.Count >= 2) return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser);
            var entry = entries[0];

            try { conn.Bind(entry.Dn, password); }
            catch (LdapException) { return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials); }

            IReadOnlyList<string> groups;
            try { groups = ExtractGroups(entry); }
            catch { return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed); }
            if (groups.Count == 0) return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);

            return LdapAuthResult.Success(username, ExtractDisplayName(entry, username), groups);
        }
        catch (StageFailure stage)
        {
            return LdapAuthResult.Fail(stage.Failure);
        }
        catch
        {
            // Fail closed: any unexpected exception is a system-side no-answer.
            return LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed);
        }
    }

Update the class-level and method XML docs: the fail-closed contract is unchanged and per-endpoint; describe the walk + stickiness (mirror the design doc's wording). Keep all existing helpers (BuildSearchAttributes, ExtractDisplayName, ExtractGroups, ToGroupShortName) untouched.

4. Append the refinement note to the design doc's Failover semantics section:

> **Refinement (Task 4):** sticky preference updates on ANY authoritative answer
> (success or user-side failure), not only on success — a DC that answers is healthy,
> and success-only stickiness would re-pay the dead primary's connect timeout on every
> mistyped password during an outage.

Step 4: Run the FULL suite (not just the new file)

Run: dotnet test ZB.MOM.WW.Auth.slnx Expected: all pass, 0 warnings — including the untouched LdapAuthServiceTests / LdapAuthServiceFailureTests / LdapEscapingTests (the behavior-preservation check for the restructure).

Step 5: Commit

git add src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs ../docs/plans/2026-08-10-ldap-backup-dc-failover-design.md
git commit -m "feat(ldap): endpoint walk with sticky-preference failover in LdapAuthService"

---

Task 5: Failover observability (ILogger) + DI wiring

Classification: standard Estimated implement time: ~4 min Parallelizable with: none (depends on Task 4)

Files:

• Modify: ZB.MOM.WW.Auth/Directory.Packages.props (add to the <!-- Extensions --> group: <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />)
• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/ZB.MOM.WW.Auth.Ldap.csproj (add <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />)
• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs
• Modify: ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.AspNetCore/ServiceCollectionExtensions.cs
• Test: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs (append)

Step 1: Write the failing test (append; include a minimal recording logger in the same file)

    private sealed class RecordingLogger : Microsoft.Extensions.Logging.ILogger<LdapAuthService>
    {
        public List<(Microsoft.Extensions.Logging.LogLevel Level, string Message)> Entries { get; } = new();
        public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
        public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
        public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId,
            TState state, Exception? exception, Func<TState, Exception?, string> formatter)
            => Entries.Add((logLevel, formatter(state, exception)));
    }

    [Fact]
    public async Task Failover_LogsWarningWithEndpointAndStage_AndInfoOnStickyChange()
    {
        var logger = new RecordingLogger();
        var factory = new SequenceLdapConnectionFactory(
            new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser());
        var svc = new LdapAuthService(Opts("dc2:3894"), factory, logger);

        await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.Contains(logger.Entries, e =>
            e.Level == Microsoft.Extensions.Logging.LogLevel.Warning
            && e.Message.Contains("dc1:636") && e.Message.Contains("connect"));
        Assert.Contains(logger.Entries, e =>
            e.Level == Microsoft.Extensions.Logging.LogLevel.Information
            && e.Message.Contains("dc2:3894"));
        // Never log secrets:
        Assert.DoesNotContain(logger.Entries, e => e.Message.Contains("svcpw") || e.Message.Contains("pw\""));
    }

    [Fact]
    public async Task HappyPath_LogsNothing()
    {
        var logger = new RecordingLogger();
        var factory = new SequenceLdapConnectionFactory(HealthyWithUser());
        var svc = new LdapAuthService(Opts("dc2"), factory, logger);

        await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);

        Assert.Empty(logger.Entries);
    }

Step 2: Run to verify failure — FAIL to compile (no 3-arg ctor).

Step 3: Implement

• Add using Microsoft.Extensions.Logging; and field private readonly ILogger<LdapAuthService>? _logger;.
• Public ctor: public LdapAuthService(LdapOptions options, ILogger<LdapAuthService>? logger = null) chaining to the internal one; internal ctor gains ILogger<LdapAuthService>? logger = null.
• In AuthenticateAgainst's catch (StageFailure stage): _logger?.LogWarning("LDAP endpoint {Endpoint} failed system-side at {Stage}; treating as no answer.", endpoint, stage.Stage); and in the final catch (bind the exception): _logger?.LogWarning(ex, "LDAP endpoint {Endpoint} failed system-side with an unexpected error.", endpoint);
• In Authenticate, at the sticky write: _logger?.LogInformation("LDAP sticky preference moved to {Endpoint}.", _endpoints[index]);
• Never log usernames' passwords or ServiceAccountPassword (assert covered by test).
• ServiceCollectionExtensions.AddZbLdapAuth: pass the logger —

        services.TryAddSingleton<ILdapAuthService>(sp =>
            new LdapAuthService(
                sp.GetRequiredService<IOptions<LdapOptions>>().Value,
                sp.GetService<Microsoft.Extensions.Logging.ILogger<LdapAuthService>>()));

(GetService, not GetRequiredService — hosts without logging still resolve. Update the surrounding "stateless" comment: the service now holds a sticky endpoint index; singleton remains correct — that state is the point.)

Step 4: Run the FULL suite

Run: dotnet test ZB.MOM.WW.Auth.slnx Expected: all pass, 0 warnings.

Step 5: Commit

git add -A
git commit -m "feat(ldap): failover observability — warn per hop, info on sticky change"

---

Task 6: GLAuth failover integration test

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 7 prep (but land before the version bump)

Files:

• Modify: ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/GLAuthIntegrationTests.cs (append one [SkippableFact])

Step 1: Add the test (mirror the file's existing env-gate + reachability-probe pattern exactly — same env vars, same Skip.IfNot / TCP probe)

    /// <summary>
    /// Failover: primary endpoint points at a closed local port (refused immediately),
    /// fallback at the real GLAuth. The login must succeed via the fallback, and a second
    /// login must also succeed (sticky preference — no re-walk through the dead primary).
    /// </summary>
    [SkippableFact]
    public async Task Authenticate_FailsOverFromDeadPrimary_ToRealGLAuth()
    {
        // <same opt-in gate, env reads, and reachability probe as the test above>

        var options = new LdapOptions
        {
            Enabled                = true,
            Server                 = "127.0.0.1",
            Port                   = 39999,          // closed port -> immediate refusal
            Transport              = LdapTransport.None,
            AllowInsecure          = true,
            SearchBase             = baseDn,
            ServiceAccountDn       = svcDn,
            ServiceAccountPassword = svcPw,
            UserNameAttribute      = userAttr,
            GroupAttribute         = "memberOf",
            ConnectionTimeoutMs    = 3000,
            FallbackServers        = new[] { $"{server}:{port}" },
        };

        var svc = new LdapAuthService(options);

        var first = await svc.AuthenticateAsync(user, pw, CancellationToken.None);
        Assert.True(first.Succeeded, $"first login failed: {first.Failure}");

        var second = await svc.AuthenticateAsync(user, pw, CancellationToken.None);
        Assert.True(second.Succeeded, $"second login failed: {second.Failure}");
    }

Step 2: Run offline — dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~GLAuthIntegrationTests" → both SKIPPED (no ZB_LDAP_IT=1).

Step 3: Run live against the shared GLAuth on the fixture host (see the file header for the full env set; server is 10.100.0.35:3893, base dc=zb,dc=local — the shared-glauth conventions):

ZB_LDAP_IT=1 ZB_LDAP_SERVER=10.100.0.35 ZB_LDAP_PORT=3893 ZB_LDAP_BASE=dc=zb,dc=local \
ZB_LDAP_SVC_DN=<svc dn> ZB_LDAP_SVC_PW=<svc pw> ZB_LDAP_USER=multi-role ZB_LDAP_PW=password \
dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~GLAuthIntegrationTests"

Expected: PASS (needs the fixture host reachable; if not, note it and continue — the unit suite carries the logic).

Step 4: Commit

git add tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/GLAuthIntegrationTests.cs
git commit -m "test(ldap): env-gated failover integration test vs live GLAuth"

---

Task 7: Version 0.2.0, pack, publish, restore-verify

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (last code task)

Files:

• Modify: ZB.MOM.WW.Auth/Directory.Build.props (<Version>0.1.5</Version> → <Version>0.2.0</Version>)

Steps:

1. Bump the version. Run the full suite one more time: dotnet test ZB.MOM.WW.Auth.slnx → all pass, 0 warnings.
2. Merge the branch: from /Users/dohertj2/Desktop/scadaproj — git checkout main && git merge --no-ff feat/auth-ldap-failover -m "feat(auth): LDAP backup-DC sticky failover — family 0.2.0" then git push origin main.
3. Pack all four packages: dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o artifacts → 4 nupkgs @ 0.2.0 (Abstractions, Ldap, ApiKeys, AspNetCore — they version together).
4. Publish to the Gitea feed (same feed/key as prior releases — the push source is https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json; credentials are already configured in the user's NuGet config from prior publishes): dotnet nuget push "artifacts/*.0.2.0.nupkg" --source <gitea feed>
5. Restore-verify from a scratch consumer (the family's standard check — see memory gitea-nuget-feed-verification): scratch console project referencing ZB.MOM.WW.Auth.Ldap 0.2.0, dotnet restore from the feed, confirm it resolves. Registration URL gotcha: verify via /api/packages/dohertj2/nuget/registration/... (NOT /nuget/v3/registration/ — that 404s for everything).

Verification gate (before claiming done): feed shows 4 packages at 0.2.0 AND scratch restore succeeds. Publishing is externally visible — if anything looks off at step 4, stop and surface it.

---

Task 8: Documentation + index truth

Classification: trivial Estimated implement time: ~4 min Parallelizable with: none (after Task 7)

Files:

• Modify: scadaproj/CLAUDE.md — Auth component row + the Auth narrative section: family is now 0.2.0 on the feed (0.1.5 still what all four apps pin); one-line description of FallbackServers sticky failover and why (NAM AD two-DC deployment).
• Modify: scadaproj/components/auth/GAPS.md — note the new capability and that app adoption (pin bump + ZB config) is a follow-up.
• Memory: update shared-lib-version-matrix.md (feed now carries Auth 0.2.0; apps still pin 0.1.5) and zb-nam-ldap-production-auth.md (failover gap CLOSED at lib level; remaining gaps: roles decision, TLS trust/1024-bit cert, app pin bumps).

Steps:

1. Make the edits. CLAUDE.md/GAPS edits go on main and get pushed (per the index rule — already on main after Task 7's merge).
2. git add -A && git commit -m "docs(index): Auth 0.2.0 — LDAP sticky backup-DC failover published" && git push origin main

---

Explicitly out of scope (per approved design)

• Credential caching (both-DCs-down survival) — rejected, YAGNI.
• Bumping the four apps' Auth pins to 0.2.0 — separate family-alignment task.
• ZB production config authoring (FallbackServers values, LDAPS trust for the 1024-bit certs) — deployment-time work tracked in ldap_setup.md.