diff --git a/docs/plans/2026-08-10-ldap-backup-dc-failover.md b/docs/plans/2026-08-10-ldap-backup-dc-failover.md new file mode 100644 index 0000000..ed8c03d --- /dev/null +++ b/docs/plans/2026-08-10-ldap-backup-dc-failover.md @@ -0,0 +1,900 @@ +# LDAP Backup-DC Failover Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Add sticky-preference multi-endpoint failover to `ZB.MOM.WW.Auth.Ldap` so a login survives a primary-DC outage (design: [`2026-08-10-ldap-backup-dc-failover-design.md`](2026-08-10-ldap-backup-dc-failover-design.md)). + +**Architecture:** One additive `LdapOptions.FallbackServers` list; `LdapAuthService` (already a singleton) walks an ordered endpoint list on system-side failures only, remembering the last endpoint that gave an authoritative answer in a `volatile int`. Per-attempt connections come from the existing `ILdapConnectionFactory` (one fresh connection per endpoint attempt — Novell connections are not safely re-connectable after failure). New optional `ILogger` makes failover visible. + +**Tech Stack:** .NET 10, Novell.Directory.Ldap.NETStandard, xunit (+ Xunit.SkippableFact for the GLAuth integration suite), central package management (`Directory.Packages.props`), family version in `Directory.Build.props`. + +**Repo/branch:** All work in `~/Desktop/scadaproj` (the Auth lib is a plain directory of this repo — do NOT `git init` anything). Create branch `feat/auth-ldap-failover` off `main`; merge back + push at the end (Task 8). + +**Working directory for all builds/tests:** `/Users/dohertj2/Desktop/scadaproj/ZB.MOM.WW.Auth` +**Test command:** `dotnet test ZB.MOM.WW.Auth.slnx` (baseline: 215 passed + 1 skipped, 0 warnings — `TreatWarningsAsErrors` conventions apply; verify baseline before starting). + +**One deliberate refinement over the design doc** (record in the design doc in Task 4): sticky preference updates on ANY authoritative answer (success *or* user-side failure like `UserNotFound`/`BadCredentials`), not only on success — a DC that answers is healthy, and updating only on success would re-pay the primary's connect timeout on every failed-password attempt during an outage. + +--- + +### Task 1: `LdapEndpoint` + `LdapEndpointParser` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 3 + +**Files:** +- Create: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/Internal/LdapEndpoint.cs` +- Test: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapEndpointParserTests.cs` (create) + +**Step 1: Write the failing tests** + +```csharp +using ZB.MOM.WW.Auth.Ldap.Internal; + +namespace ZB.MOM.WW.Auth.Ldap.Tests; + +public class LdapEndpointParserTests +{ + [Theory] + [InlineData("dc2.example.com", 636, "dc2.example.com", 636)] // bare host -> default port + [InlineData("dc2.example.com:3894", 636, "dc2.example.com", 3894)] // explicit port wins + [InlineData(" dc2.example.com : 3894 ", 636, "dc2.example.com", 3894)] // whitespace tolerated + public void TryParse_ValidEntry_Succeeds(string entry, int defaultPort, string host, int port) + { + Assert.True(LdapEndpointParser.TryParse(entry, defaultPort, out var ep, out _)); + Assert.Equal(host, ep.Host); + Assert.Equal(port, ep.Port); + } + + [Theory] + [InlineData("")] // empty + [InlineData(" ")] // whitespace + [InlineData(":636")] // empty host + [InlineData("dc2:0")] // port below range + [InlineData("dc2:65536")] // port above range + [InlineData("dc2:abc")] // non-numeric port + [InlineData("2001:db8::1:636")] // IPv6 literal (unsupported, >1 colon) + public void TryParse_InvalidEntry_Fails(string entry) + { + Assert.False(LdapEndpointParser.TryParse(entry, 636, out _, out var error)); + Assert.NotEmpty(error); + } + + [Fact] + public void Endpoint_ToString_IsHostColonPort() => + Assert.Equal("dc2:3894", new LdapEndpoint("dc2", 3894).ToString()); +} +``` + +**Step 2: Run to verify failure** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapEndpointParserTests"` +Expected: FAIL to compile (`LdapEndpoint` not defined). + +**Step 3: Implement** + +```csharp +namespace ZB.MOM.WW.Auth.Ldap.Internal; + +/// A single LDAP server endpoint resolved from configuration. +internal sealed record LdapEndpoint(string Host, int Port) +{ + public override string ToString() => $"{Host}:{Port}"; +} + +/// +/// Parses FallbackServers entries: "host" (adopts the caller's default port) +/// or "host:port". IPv6 literals are not supported — use a DNS name. +/// +internal static class LdapEndpointParser +{ + public static bool TryParse(string entry, int defaultPort, out LdapEndpoint endpoint, out string error) + { + endpoint = null!; + error = ""; + + if (string.IsNullOrWhiteSpace(entry)) + { + error = "entry is empty or whitespace"; + return false; + } + + var parts = entry.Trim().Split(':'); + if (parts.Length > 2) + { + error = $"entry \"{entry.Trim()}\" contains more than one ':' " + + "(IPv6 literals are not supported — use a DNS name)"; + return false; + } + + var host = parts[0].Trim(); + if (host.Length == 0) + { + error = $"entry \"{entry.Trim()}\" has an empty host"; + return false; + } + + var port = defaultPort; + if (parts.Length == 2 && (!int.TryParse(parts[1].Trim(), out port) || port is < 1 or > 65535)) + { + error = $"entry \"{entry.Trim()}\" has an invalid port (must be 1–65535)"; + return false; + } + + endpoint = new LdapEndpoint(host, port); + return true; + } +} +``` + +**Step 4: Run tests, verify pass** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapEndpointParserTests"` +Expected: 11 PASS. + +**Step 5: Commit** + +```bash +git add src/ZB.MOM.WW.Auth.Ldap/Internal/LdapEndpoint.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapEndpointParserTests.cs +git commit -m "feat(ldap): LdapEndpoint + FallbackServers entry parser" +``` + +--- + +### Task 2: `LdapOptions.FallbackServers` + validator rules + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 3 (NOT Task 1 — uses the parser) + +**Files:** +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs` (add property after `ConnectionTimeoutMs`, line ~20) +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs` (new check after the `ServiceAccountDn` check) +- Test: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapOptionsValidatorTests.cs` (append) + +**Step 1: Write the failing tests** (append to `LdapOptionsValidatorTests` — match its terse `Opts() with {}` style) + +```csharp + [Fact] + public void Validator_Succeeds_WithValidFallbackServers() => + Assert.False(new LdapOptionsValidator() + .Validate(null, Opts() with { FallbackServers = new[] { "dc2", "dc3:3894" } }) + .Failed); + + [Theory] + [InlineData("")] + [InlineData(":636")] + [InlineData("dc2:0")] + [InlineData("dc2:notaport")] + public void Validator_Fails_OnMalformedFallbackServer(string entry) + { + var result = new LdapOptionsValidator() + .Validate(null, Opts() with { FallbackServers = new[] { entry } }); + + Assert.True(result.Failed); + Assert.Contains(nameof(LdapOptions.FallbackServers), result.FailureMessage); + } + + [Fact] + public void Validator_Skips_FallbackServers_WhenDisabled() => + Assert.False(new LdapOptionsValidator() + .Validate(null, Opts() with { Enabled = false, FallbackServers = new[] { ":::" } }) + .Failed); +``` + +**Step 2: Run to verify failure** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapOptionsValidatorTests"` +Expected: FAIL to compile (`FallbackServers` not defined). + +**Step 3: Implement** + +In `LdapContracts.cs`, add to `LdapOptions` after `ConnectionTimeoutMs`: + +```csharp + /// + /// Ordered fallback LDAP endpoints tried when the primary (and any earlier fallback) + /// fails with a system-side error (connect/TLS, service-account bind, or search). + /// Each entry is "host" (adopts ) or "host:port". + /// Empty (the default) disables failover entirely — single-endpoint behavior is + /// unchanged. The TLS validation policy ( + /// / ) applies to every endpoint. Endpoint preference is + /// sticky: the service keeps using the last endpoint that answered until it fails. + /// + public IReadOnlyList FallbackServers { get; init; } = Array.Empty(); +``` + +In `LdapOptionsValidator.Validate`, after the `ServiceAccountDn` check (before the final `return`): + +```csharp + foreach (var entry in options.FallbackServers) + { + if (!Internal.LdapEndpointParser.TryParse(entry, options.Port, out _, out var error)) + { + return ValidateOptionsResult.Fail( + $"{nameof(LdapOptions.FallbackServers)} contains an invalid entry — {error}. " + + "Each entry must be \"host\" or \"host:port\" (port 1–65535)."); + } + } +``` + +Also extend the class `` bullet list with a fifth condition (fallback entries must parse). + +**Step 4: Run tests, verify pass** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapOptionsValidatorTests"` +Expected: all PASS (existing + 6 new). + +**Step 5: Commit** + +```bash +git add src/ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs src/ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapOptionsValidatorTests.cs +git commit -m "feat(ldap): FallbackServers option + boot-time validation" +``` + +--- + +### Task 3: `SequenceLdapConnectionFactory` test double + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 1, Task 2 + +The existing `FakeLdapConnectionFactory` returns ONE shared fake — useless for failover tests, where each endpoint attempt gets a fresh connection and the fakes must differ per attempt. Do NOT modify `FakeLdapConnection` (its `BindAttempts == 1` service-bind heuristic is per-instance and stays valid because the walk creates a fresh connection per attempt). + +**Files:** +- Create: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/SequenceLdapConnectionFactory.cs` + +**Step 1: Implement (with inline smoke test, mirroring FakeLdapConnection.cs's pattern)** + +```csharp +using ZB.MOM.WW.Auth.Ldap.Internal; + +namespace ZB.MOM.WW.Auth.Ldap.Tests; + +/// +/// Test double for that hands out pre-scripted +/// instances in order, one per call — +/// exactly matching the failover walk, which opens a fresh connection per endpoint attempt. +/// Records every created connection for inspection. When the script runs out, returns a +/// bare unscripted fake (which yields user-not-found) so a miscounted test fails loudly +/// rather than reusing a stale fake. +/// +internal sealed class SequenceLdapConnectionFactory : ILdapConnectionFactory +{ + private readonly Queue _script; + + public SequenceLdapConnectionFactory(params FakeLdapConnection[] fakes) + => _script = new Queue(fakes); + + /// Every connection handed out, in creation order (scripted or overflow). + public List Created { get; } = new(); + + public ILdapConnection Create() + { + var fake = _script.Count > 0 ? _script.Dequeue() : new FakeLdapConnection(); + Created.Add(fake); + return fake; + } +} + +public class SequenceLdapConnectionFactorySmokeTests +{ + [Fact] + public void Create_ReturnsScriptedFakesInOrder_ThenOverflows() + { + var first = new FakeLdapConnection().ThrowOnConnect(); + var second = new FakeLdapConnection(); + var factory = new SequenceLdapConnectionFactory(first, second); + + Assert.Same(first, factory.Create()); + Assert.Same(second, factory.Create()); + Assert.NotSame(second, factory.Create()); // overflow: fresh bare fake + Assert.Equal(3, factory.Created.Count); + } +} +``` + +**Step 2: Run, verify pass** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~SequenceLdapConnectionFactorySmokeTests"` +Expected: 1 PASS. + +**Step 3: Commit** + +```bash +git add tests/ZB.MOM.WW.Auth.Ldap.Tests/SequenceLdapConnectionFactory.cs +git commit -m "test(ldap): sequence connection factory for failover tests" +``` + +--- + +### Task 4: Endpoint walk + sticky preference in `LdapAuthService` + +**Classification:** high-risk (auth-critical logic; the fail-closed contract must survive the restructure) +**Estimated implement time:** ~5 min +**Parallelizable with:** none (depends on Tasks 1–3) + +**Files:** +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs` (restructure `Authenticate`; keep every stage's behavior byte-identical inside the per-endpoint attempt) +- Test: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs` (create) +- Modify (doc note only): `scadaproj/docs/plans/2026-08-10-ldap-backup-dc-failover-design.md` — record the sticky-on-any-authoritative-answer refinement. + +NOTE: the `ILogger` parameter arrives in Task 5 — in this task the service compiles WITHOUT logging (leave the `_logger?` lines out; Task 5 adds them). Existing tests in `LdapAuthServiceTests` / `LdapAuthServiceFailureTests` must pass unchanged — they use single-endpoint options and the shared-instance `FakeLdapConnectionFactory`, which still works because a single-endpoint walk calls `Create()` once. + +**Step 1: Write the failing tests** + +```csharp +using ZB.MOM.WW.Auth.Abstractions.Ldap; + +namespace ZB.MOM.WW.Auth.Tests; // match the existing LdapAuthServiceTests namespace — check and mirror it + +public class LdapAuthServiceFailoverTests +{ + private static LdapOptions Opts(params string[] fallbacks) => new() + { + Enabled = true, + Server = "dc1", + Port = 636, + Transport = LdapTransport.Ldaps, + SearchBase = "dc=x", + ServiceAccountDn = "cn=svc,dc=x", + ServiceAccountPassword = "svcpw", + FallbackServers = fallbacks, + }; + + private static FakeLdapConnection HealthyWithUser() => + new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=ops,dc=x" }); + + // ---- failover triggers (system-side) ---- + + [Fact] + public async Task ConnectFailure_FailsOverToFallback_AndSucceeds() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser()); + var svc = new LdapAuthService(Opts("dc2:3894"), factory); + + var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.True(result.Succeeded); + Assert.Equal(2, factory.Created.Count); + Assert.Equal("dc1", factory.Created[0].ConnectArgs!.Value.Host); + Assert.Equal("dc2", factory.Created[1].ConnectArgs!.Value.Host); + Assert.Equal(3894, factory.Created[1].ConnectArgs!.Value.Port); + } + + [Fact] + public async Task ServiceBindFailure_FailsOverToFallback() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnServiceBind(), HealthyWithUser()); + var svc = new LdapAuthService(Opts("dc2"), factory); + + var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.True(result.Succeeded); + Assert.Equal(636, factory.Created[1].ConnectArgs!.Value.Port); // bare host adopts Port + } + + // ---- user-side results NEVER fail over ---- + + [Fact] + public async Task BadUserCredentials_DoesNotFailOver() + { + var factory = new SequenceLdapConnectionFactory( + HealthyWithUser().ThrowOnUserBind()); + var svc = new LdapAuthService(Opts("dc2"), factory); + + var result = await svc.AuthenticateAsync("alice", "wrong", CancellationToken.None); + + Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure); + Assert.Single(factory.Created); + } + + [Fact] + public async Task UserNotFound_DoesNotFailOver() + { + var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection()); + var svc = new LdapAuthService(Opts("dc2"), factory); + + var result = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None); + + Assert.Equal(LdapAuthFailure.UserNotFound, result.Failure); + Assert.Single(factory.Created); + } + + [Fact] + public async Task GroupLookupFailed_DoesNotFailOver() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", Array.Empty())); + var svc = new LdapAuthService(Opts("dc2"), factory); + + var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure); + Assert.Single(factory.Created); + } + + // ---- exhaustion ---- + + [Fact] + public async Task AllEndpointsDown_ReturnsServiceAccountBindFailed() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), + new FakeLdapConnection().ThrowOnConnect()); + var svc = new LdapAuthService(Opts("dc2"), factory); + + var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure); + Assert.Equal(2, factory.Created.Count); + } + + // ---- stickiness ---- + + [Fact] + public async Task StickyPreference_SecondLoginStartsAtLastGoodEndpoint() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead + HealthyWithUser(), // login 1 @ dc2: answers + HealthyWithUser()); // login 2: should go straight to dc2 + var svc = new LdapAuthService(Opts("dc2"), factory); + + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + + Assert.Equal(3, factory.Created.Count); + Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host); + } + + [Fact] + public async Task StickyPreference_DriftsBackToPrimary_WhenPreferredFails() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead + HealthyWithUser(), // login 1 @ dc2: answers -> preferred + new FakeLdapConnection().ThrowOnConnect(), // login 2 @ dc2: now dead + HealthyWithUser(), // login 2 @ dc1 (declared order): recovered + HealthyWithUser()); // login 3: should start at dc1 again + var svc = new LdapAuthService(Opts("dc2"), factory); + + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + + Assert.Equal(5, factory.Created.Count); + Assert.Equal("dc1", factory.Created[3].ConnectArgs!.Value.Host); + Assert.Equal("dc1", factory.Created[4].ConnectArgs!.Value.Host); + } + + [Fact] + public async Task StickyPreference_UpdatesOnUserSideAnswer_NotOnlySuccess() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead + new FakeLdapConnection(), // login 1 @ dc2: answers UserNotFound + HealthyWithUser()); // login 2: should start at dc2 + var svc = new LdapAuthService(Opts("dc2"), factory); + + var first = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None); + Assert.Equal(LdapAuthFailure.UserNotFound, first.Failure); + + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host); + } + + // ---- back-compat ---- + + [Fact] + public async Task NoFallbacks_SingleAttempt_UnchangedBehavior() + { + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect()); + var svc = new LdapAuthService(Opts(/* no fallbacks */), factory); + + var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure); + Assert.Single(factory.Created); + } + + [Fact] + public async Task MalformedFallbackEntry_IsSkippedDefensively() + { + // The validator rejects this at boot; a service constructed without validation + // must degrade to the parseable subset, not throw. + var factory = new SequenceLdapConnectionFactory(HealthyWithUser()); + var svc = new LdapAuthService(Opts(":::"), factory); + + Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded); + Assert.Single(factory.Created); + } +} +``` + +Before writing: open `LdapAuthServiceTests.cs`, mirror its exact namespace and the internal-visibility usage (the test project has `InternalsVisibleTo` — `LdapAuthService`'s internal ctor and the fakes are reachable). + +**Step 2: Run to verify failure** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~LdapAuthServiceFailoverTests"` +Expected: FAIL — walk/stickiness not implemented (connect-failure tests fail with a single `Created` entry; sticky tests fail). + +**Step 3: Implement** — restructure `LdapAuthService`: + +1. New fields + endpoint construction: + +```csharp + private readonly IReadOnlyList _endpoints; + private int _preferredIndex; // index into _endpoints; volatile access; benign races accepted +``` + +In BOTH constructors' shared path (the internal one; the public one chains to it), set +`_endpoints = BuildEndpoints(options);` and add: + +```csharp + /// + /// Primary first, then fallbacks in declared order. The validator rejects malformed + /// entries at boot; unparseable entries are skipped defensively here so a service + /// constructed without options validation degrades to the parseable subset. + /// + private static IReadOnlyList BuildEndpoints(LdapOptions options) + { + var endpoints = new List { new(options.Server, options.Port) }; + foreach (var entry in options.FallbackServers) + { + if (LdapEndpointParser.TryParse(entry, options.Port, out var ep, out _)) + endpoints.Add(ep); + } + return endpoints; + } +``` + +2. `Authenticate` becomes the walk (steps 1–3 — gate, null-username, trim — stay as-is above it): + +```csharp + private LdapAuthResult Authenticate(string username, string password) + { + if (!_options.Enabled) + return LdapAuthResult.Fail(LdapAuthFailure.Disabled); + if (string.IsNullOrWhiteSpace(username)) + return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials); + username = username.Trim(); + + var preferred = Volatile.Read(ref _preferredIndex); + if ((uint)preferred >= (uint)_endpoints.Count) + preferred = 0; // defensive: index can never exceed the immutable list, but fail safe + + var lastSystemSideFailure = LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed); + foreach (var index in WalkOrder(_endpoints.Count, preferred)) + { + var result = AuthenticateAgainst(_endpoints[index], username, password); + + // ServiceAccountBindFailed is the (sole) system-side bucket: the endpoint gave no + // authoritative answer — try the next one. Every other outcome (success, or a + // user-side failure like UserNotFound/BadCredentials) came from a healthy + // directory and is final. + if (result.Failure == LdapAuthFailure.ServiceAccountBindFailed) + { + lastSystemSideFailure = result; + continue; + } + + // Sticky preference: any authoritative answer proves the endpoint healthy — + // updating only on Succeeded would re-pay the dead primary's connect timeout + // on every mistyped password during an outage. + if (index != preferred) + Volatile.Write(ref _preferredIndex, index); + + return result; + } + + return lastSystemSideFailure; + } + + /// Preferred endpoint first, then the rest in declared order. + private static IEnumerable WalkOrder(int count, int preferred) + { + yield return preferred; + for (var i = 0; i < count; i++) + { + if (i != preferred) + yield return i; + } + } +``` + +3. `AuthenticateAgainst(LdapEndpoint endpoint, ...)` is the CURRENT stages 4–10 verbatim, with three mechanical changes: (a) `conn.Connect(endpoint.Host, endpoint.Port, ...)` instead of `_options.Server/_options.Port`; (b) the user-bind and group-extraction failures return directly instead of throwing `StageFailure` (equivalent, simpler); (c) `StageFailure` gains a `string Stage` member (`"connect"` / `"service-bind"` / `"search"`) consumed by Task 5's logging — carry it now so Task 5 is additive: + +```csharp + private LdapAuthResult AuthenticateAgainst(LdapEndpoint endpoint, string username, string password) + { + try + { + using var conn = _connectionFactory.Create(); + + try + { + conn.Connect(endpoint.Host, endpoint.Port, _options.Transport, + _options.AllowInsecure, _options.ConnectionTimeoutMs, + _options.ServerCertificateValidationCallback); + } + catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "connect"); } + + try { conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword); } + catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "service-bind"); } + + IReadOnlyList entries; + try + { + var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})"; + entries = conn.Search(_options.SearchBase, filter, BuildSearchAttributes()); + } + catch (LdapException) { throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "search"); } + + if (entries.Count == 0) return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound); + if (entries.Count >= 2) return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser); + var entry = entries[0]; + + try { conn.Bind(entry.Dn, password); } + catch (LdapException) { return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials); } + + IReadOnlyList groups; + try { groups = ExtractGroups(entry); } + catch { return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed); } + if (groups.Count == 0) return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed); + + return LdapAuthResult.Success(username, ExtractDisplayName(entry, username), groups); + } + catch (StageFailure stage) + { + return LdapAuthResult.Fail(stage.Failure); + } + catch + { + // Fail closed: any unexpected exception is a system-side no-answer. + return LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed); + } + } +``` + +Update the class-level and method XML docs: the fail-closed contract is unchanged and per-endpoint; describe the walk + stickiness (mirror the design doc's wording). Keep all existing helpers (`BuildSearchAttributes`, `ExtractDisplayName`, `ExtractGroups`, `ToGroupShortName`) untouched. + +4. Append the refinement note to the design doc's *Failover semantics* section: + +```markdown +> **Refinement (Task 4):** sticky preference updates on ANY authoritative answer +> (success or user-side failure), not only on success — a DC that answers is healthy, +> and success-only stickiness would re-pay the dead primary's connect timeout on every +> mistyped password during an outage. +``` + +**Step 4: Run the FULL suite (not just the new file)** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx` +Expected: all pass, 0 warnings — including the untouched `LdapAuthServiceTests` / `LdapAuthServiceFailureTests` / `LdapEscapingTests` (the behavior-preservation check for the restructure). + +**Step 5: Commit** + +```bash +git add src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs ../docs/plans/2026-08-10-ldap-backup-dc-failover-design.md +git commit -m "feat(ldap): endpoint walk with sticky-preference failover in LdapAuthService" +``` + +--- + +### Task 5: Failover observability (`ILogger`) + DI wiring + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none (depends on Task 4) + +**Files:** +- Modify: `ZB.MOM.WW.Auth/Directory.Packages.props` (add to the `` group: ``) +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/ZB.MOM.WW.Auth.Ldap.csproj` (add ``) +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs` +- Modify: `ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.AspNetCore/ServiceCollectionExtensions.cs` +- Test: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs` (append) + +**Step 1: Write the failing test** (append; include a minimal recording logger in the same file) + +```csharp + private sealed class RecordingLogger : Microsoft.Extensions.Logging.ILogger + { + public List<(Microsoft.Extensions.Logging.LogLevel Level, string Message)> Entries { get; } = new(); + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true; + public void Log(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, + TState state, Exception? exception, Func formatter) + => Entries.Add((logLevel, formatter(state, exception))); + } + + [Fact] + public async Task Failover_LogsWarningWithEndpointAndStage_AndInfoOnStickyChange() + { + var logger = new RecordingLogger(); + var factory = new SequenceLdapConnectionFactory( + new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser()); + var svc = new LdapAuthService(Opts("dc2:3894"), factory, logger); + + await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.Contains(logger.Entries, e => + e.Level == Microsoft.Extensions.Logging.LogLevel.Warning + && e.Message.Contains("dc1:636") && e.Message.Contains("connect")); + Assert.Contains(logger.Entries, e => + e.Level == Microsoft.Extensions.Logging.LogLevel.Information + && e.Message.Contains("dc2:3894")); + // Never log secrets: + Assert.DoesNotContain(logger.Entries, e => e.Message.Contains("svcpw") || e.Message.Contains("pw\"")); + } + + [Fact] + public async Task HappyPath_LogsNothing() + { + var logger = new RecordingLogger(); + var factory = new SequenceLdapConnectionFactory(HealthyWithUser()); + var svc = new LdapAuthService(Opts("dc2"), factory, logger); + + await svc.AuthenticateAsync("alice", "pw", CancellationToken.None); + + Assert.Empty(logger.Entries); + } +``` + +**Step 2: Run to verify failure** — FAIL to compile (no 3-arg ctor). + +**Step 3: Implement** + +- Add `using Microsoft.Extensions.Logging;` and field `private readonly ILogger? _logger;`. +- Public ctor: `public LdapAuthService(LdapOptions options, ILogger? logger = null)` chaining to the internal one; internal ctor gains `ILogger? logger = null`. +- In `AuthenticateAgainst`'s `catch (StageFailure stage)`: + `_logger?.LogWarning("LDAP endpoint {Endpoint} failed system-side at {Stage}; treating as no answer.", endpoint, stage.Stage);` + and in the final `catch` (bind the exception): + `_logger?.LogWarning(ex, "LDAP endpoint {Endpoint} failed system-side with an unexpected error.", endpoint);` +- In `Authenticate`, at the sticky write: + `_logger?.LogInformation("LDAP sticky preference moved to {Endpoint}.", _endpoints[index]);` +- Never log usernames' passwords or `ServiceAccountPassword` (assert covered by test). +- `ServiceCollectionExtensions.AddZbLdapAuth`: pass the logger — + +```csharp + services.TryAddSingleton(sp => + new LdapAuthService( + sp.GetRequiredService>().Value, + sp.GetService>())); +``` + +(`GetService`, not `GetRequiredService` — hosts without logging still resolve. Update the surrounding "stateless" comment: the service now holds a sticky endpoint index; singleton remains correct — that state is the point.) + +**Step 4: Run the FULL suite** + +Run: `dotnet test ZB.MOM.WW.Auth.slnx` +Expected: all pass, 0 warnings. + +**Step 5: Commit** + +```bash +git add -A +git commit -m "feat(ldap): failover observability — warn per hop, info on sticky change" +``` + +--- + +### Task 6: GLAuth failover integration test + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 7 prep (but land before the version bump) + +**Files:** +- Modify: `ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/GLAuthIntegrationTests.cs` (append one `[SkippableFact]`) + +**Step 1: Add the test** (mirror the file's existing env-gate + reachability-probe pattern exactly — same env vars, same `Skip.IfNot` / TCP probe) + +```csharp + /// + /// Failover: primary endpoint points at a closed local port (refused immediately), + /// fallback at the real GLAuth. The login must succeed via the fallback, and a second + /// login must also succeed (sticky preference — no re-walk through the dead primary). + /// + [SkippableFact] + public async Task Authenticate_FailsOverFromDeadPrimary_ToRealGLAuth() + { + // + + var options = new LdapOptions + { + Enabled = true, + Server = "127.0.0.1", + Port = 39999, // closed port -> immediate refusal + Transport = LdapTransport.None, + AllowInsecure = true, + SearchBase = baseDn, + ServiceAccountDn = svcDn, + ServiceAccountPassword = svcPw, + UserNameAttribute = userAttr, + GroupAttribute = "memberOf", + ConnectionTimeoutMs = 3000, + FallbackServers = new[] { $"{server}:{port}" }, + }; + + var svc = new LdapAuthService(options); + + var first = await svc.AuthenticateAsync(user, pw, CancellationToken.None); + Assert.True(first.Succeeded, $"first login failed: {first.Failure}"); + + var second = await svc.AuthenticateAsync(user, pw, CancellationToken.None); + Assert.True(second.Succeeded, $"second login failed: {second.Failure}"); + } +``` + +**Step 2: Run offline** — `dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~GLAuthIntegrationTests"` → both SKIPPED (no `ZB_LDAP_IT=1`). + +**Step 3: Run live** against the shared GLAuth on the fixture host (see the file header for the full env set; server is `10.100.0.35:3893`, base `dc=zb,dc=local` — the shared-glauth conventions): + +```bash +ZB_LDAP_IT=1 ZB_LDAP_SERVER=10.100.0.35 ZB_LDAP_PORT=3893 ZB_LDAP_BASE=dc=zb,dc=local \ +ZB_LDAP_SVC_DN= ZB_LDAP_SVC_PW= ZB_LDAP_USER=multi-role ZB_LDAP_PW=password \ +dotnet test ZB.MOM.WW.Auth.slnx --filter "FullyQualifiedName~GLAuthIntegrationTests" +``` + +Expected: PASS (needs the fixture host reachable; if not, note it and continue — the unit suite carries the logic). + +**Step 4: Commit** + +```bash +git add tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/GLAuthIntegrationTests.cs +git commit -m "test(ldap): env-gated failover integration test vs live GLAuth" +``` + +--- + +### Task 7: Version 0.2.0, pack, publish, restore-verify + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (last code task) + +**Files:** +- Modify: `ZB.MOM.WW.Auth/Directory.Build.props` (`0.1.5` → `0.2.0`) + +**Steps:** + +1. Bump the version. Run the full suite one more time: `dotnet test ZB.MOM.WW.Auth.slnx` → all pass, 0 warnings. +2. Merge the branch: from `/Users/dohertj2/Desktop/scadaproj` — `git checkout main && git merge --no-ff feat/auth-ldap-failover -m "feat(auth): LDAP backup-DC sticky failover — family 0.2.0"` then `git push origin main`. +3. Pack all four packages: `dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o artifacts` → 4 nupkgs @ 0.2.0 (Abstractions, Ldap, ApiKeys, AspNetCore — they version together). +4. Publish to the Gitea feed (same feed/key as prior releases — the push source is `https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json`; credentials are already configured in the user's NuGet config from prior publishes): + `dotnet nuget push "artifacts/*.0.2.0.nupkg" --source ` +5. Restore-verify from a scratch consumer (the family's standard check — see memory `gitea-nuget-feed-verification`): scratch console project referencing `ZB.MOM.WW.Auth.Ldap 0.2.0`, `dotnet restore` from the feed, confirm it resolves. Registration URL gotcha: verify via `/api/packages/dohertj2/nuget/registration/...` (NOT `/nuget/v3/registration/` — that 404s for everything). + +**Verification gate (before claiming done):** feed shows 4 packages at 0.2.0 AND scratch restore succeeds. Publishing is externally visible — if anything looks off at step 4, stop and surface it. + +--- + +### Task 8: Documentation + index truth + +**Classification:** trivial +**Estimated implement time:** ~4 min +**Parallelizable with:** none (after Task 7) + +**Files:** +- Modify: `scadaproj/CLAUDE.md` — Auth component row + the Auth narrative section: family is now `0.2.0` on the feed (`0.1.5` still what all four apps pin); one-line description of `FallbackServers` sticky failover and why (NAM AD two-DC deployment). +- Modify: `scadaproj/components/auth/GAPS.md` — note the new capability and that app adoption (pin bump + ZB config) is a follow-up. +- Memory: update `shared-lib-version-matrix.md` (feed now carries Auth 0.2.0; apps still pin 0.1.5) and `zb-nam-ldap-production-auth.md` (failover gap CLOSED at lib level; remaining gaps: roles decision, TLS trust/1024-bit cert, app pin bumps). + +**Steps:** + +1. Make the edits. CLAUDE.md/GAPS edits go on `main` and get pushed (per the index rule — already on `main` after Task 7's merge). +2. `git add -A && git commit -m "docs(index): Auth 0.2.0 — LDAP sticky backup-DC failover published" && git push origin main` + +--- + +## Explicitly out of scope (per approved design) + +- Credential caching (both-DCs-down survival) — rejected, YAGNI. +- Bumping the four apps' Auth pins to 0.2.0 — separate family-alignment task. +- ZB production config authoring (`FallbackServers` values, LDAPS trust for the 1024-bit certs) — deployment-time work tracked in `ldap_setup.md`. diff --git a/docs/plans/2026-08-10-ldap-backup-dc-failover.md.tasks.json b/docs/plans/2026-08-10-ldap-backup-dc-failover.md.tasks.json new file mode 100644 index 0000000..a4c03a2 --- /dev/null +++ b/docs/plans/2026-08-10-ldap-backup-dc-failover.md.tasks.json @@ -0,0 +1,14 @@ +{ + "planPath": "docs/plans/2026-08-10-ldap-backup-dc-failover.md", + "tasks": [ + {"id": 1, "subject": "Task 1: LdapEndpoint + LdapEndpointParser", "status": "pending"}, + {"id": 2, "subject": "Task 2: LdapOptions.FallbackServers + validator rules", "status": "pending", "blockedBy": [1]}, + {"id": 3, "subject": "Task 3: SequenceLdapConnectionFactory test double", "status": "pending"}, + {"id": 4, "subject": "Task 4: Endpoint walk + sticky preference in LdapAuthService", "status": "pending", "blockedBy": [1, 2, 3]}, + {"id": 5, "subject": "Task 5: Failover observability (ILogger) + DI wiring", "status": "pending", "blockedBy": [4]}, + {"id": 6, "subject": "Task 6: GLAuth failover integration test", "status": "pending", "blockedBy": [5]}, + {"id": 7, "subject": "Task 7: Version 0.2.0, pack, publish, restore-verify", "status": "pending", "blockedBy": [6]}, + {"id": 8, "subject": "Task 8: Documentation + index truth", "status": "pending", "blockedBy": [7]} + ], + "lastUpdated": "2026-08-10" +}