scadaproj/docs/plans/2026-08-10-ldap-backup-dc-failover-design.md

LDAP backup-DC failover — design (ZB.MOM.WW.Auth 0.2.0)

Date: 2026-08-10 Status: Approved Motivation: Production servers at ZB authenticate against the Zimmer NAM AD (ldap_setup.md), which has two domain controllers (uswap-vdc-nam1 / uswap-vdc-nam2). ZB.MOM.WW.Auth.Ldap takes a single Server, so a primary-DC outage blocks every interactive login family-wide (API keys unaffected). The working Ignition gateway has primary + secondary DC failover; this brings the shared library to parity on that axis.

Decisions (made with the user, 2026-08-10)

1. Sticky preference failover — try the last-known-good DC first; on failure walk the ordered list and remember whichever worked. Mirrors the Secrets 0.5.0 FallbackEndpoints sticky/no-flapping precedent. Rejected: always-primary-first (pays the full connection timeout on every login during a primary outage), sticky + probe-back (YAGNI for two DCs).
2. Failover-only — no Ignition-style credential cache. A both-DCs-down event is rare and correlated with wider outages; API keys already survive it; a credential cache adds credential-at-rest surface. Possible future follow-up.
3. Approach A: service-level loop — the walk lives in LdapAuthService, which already owns failure classification (user-fault vs system-fault). Rejected: connection-factory-level (only sees Connect; a DC that accepts TCP+TLS but fails service-bind/search would never fail over), DNS-level nam.zimmer.com (DC certs' SANs carry only their own FQDN → LDAPS hostname validation breaks; no stickiness or timeout control).

Probe evidence (2026-08-10, recorded in ldap_setup.md)

Both DCs serve LDAPS (636, TLSv1.3) and StartTLS (389); anonymous rootDSE over LDAPS returns result: 0 Success on both. Certs: per-host FQDN SANs, issued by zimmer-issueca1 ← zimmer-rootca (internal CA, root not served), valid to 2027-03. Caveat: both certs are RSA 1024-bit — OpenSSL at SECLEVEL=2 (Debian-based .NET containers) rejects the handshake (EE certificate key too weak); Windows/Schannel hosts accept. Container deployments need a lowered SECLEVEL, a validation-callback path, or (right fix) a reissued 2048-bit DC cert template.

Config surface

One additive field on LdapOptions (ZB.MOM.WW.Auth.Abstractions):

/// <summary>Ordered fallback LDAP endpoints, "host" or "host:port"; a bare host uses <see cref="Port"/>.</summary>
public IReadOnlyList<string> FallbackServers { get; init; } = Array.Empty<string>();

"Ldap": {
  "Server": "uswap-vdc-nam1.nam.zimmer.com",
  "Port": 636,
  "Transport": "Ldaps",
  "FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com" ]
}

• Empty (the default) ⇒ behavior byte-identical to today: single endpoint, no loop, no new state.
• ServerCertificateValidationCallback and AllowInsecure apply to every endpoint (one CA family — verified by the probe).
• Validator (LdapOptionsValidator, only when Enabled): each entry must have a non-empty host; an explicit port must parse as 1–65535. Malformed entries fail boot with a field-naming message in the validator's existing style. Duplicates are permitted (harmless; de-duplication is not the validator's job).

Failover semantics

Inside LdapAuthService.Authenticate, an endpoint walk over an internal LdapEndpoint(string Host, int Port) list = Server:Port + parsed FallbackServers, in declared order:

• Walk order: sticky-preferred endpoint first, then the remaining endpoints in declared order (skipping the preferred). On success at endpoint i, preference := i.
• Only system-side failures advance the walk: connect/TLS-upgrade failure, service-account bind failure, search failure, and any unexpected exception inside an attempt. User-side results return immediately, never fail over: UserNotFound, AmbiguousUser, BadCredentials (user bind), GroupLookupFailed — those are answers from a healthy directory.
• Exhaustion: all endpoints failed system-side ⇒ return the last failure (maps to ServiceAccountBindFailed, as today — no LdapAuthFailure enum change, no Abstractions contract change beyond the new option).
• Fail-closed contract preserved: AuthenticateAsync still never throws; each per-endpoint attempt is individually guarded.
• Documented trade-offs:
  • A genuinely wrong service-account password tries every DC per login (fails fast at bind — no timeout cost; 2 endpoints in practice).
  • Worst-case login latency while the preferred DC is down = ConnectionTimeoutMs × walk position, paid once — stickiness then moves preference to the survivor.
  • Sticky means the primary stays unused after recovery until the current preferred fails; when it does, the walk tries declared order again, so preference drifts back to the primary naturally.

Refinement (Task 4): sticky preference updates on ANY authoritative answer (success or user-side failure), not only on success — a DC that answers is healthy, and success-only stickiness would re-pay the dead primary's connect timeout on every mistyped password during an outage. Corollary, also implemented: an exhausted walk (every endpoint system-side-down) leaves the preference untouched, since a total outage says nothing about which endpoint to prefer once one recovers.

Stickiness state

volatile int _preferredIndex on the singleton LdapAuthService (registered TryAddSingleton in ZB.MOM.WW.Auth.AspNetCore). Benign races tolerated — worst case is one redundant failed attempt. No cross-process or persisted state.

Observability

New optional ILogger<LdapAuthService>? constructor parameter (null default — existing constructors keep compiling; adds only Microsoft.Extensions.Logging.Abstractions to ZB.MOM.WW.Auth.Ldap). The AspNetCore registration passes the host logger. Log points:

• Warning per failover hop: endpoint, stage that failed, next endpoint.
• Information on sticky-preference change (old → new endpoint).
• Nothing on the happy path; never log credentials or passwords.

Rationale: silent failover hides a dead primary DC indefinitely — the feature working correctly must be visible to operators.

Testing

• Unit (FakeLdapConnection seam): failover triggered by each system-side stage (connect, service bind, search, unexpected exception); NO failover on each user-side result; stickiness updates on success and drifts back after the preferred fails; exhaustion returns ServiceAccountBindFailed; empty FallbackServers short-circuits to today's exact behavior; endpoint parsing (host, host:port, malformed).
• Validator tests: new cases in LdapOptionsValidatorTests style.
• Integration (env-gated GLAuth suite): primary pointed at a closed local port + fallback at the real GLAuth ⇒ login succeeds, preference sticks, and a second login does not retry the dead primary. Fast offline too — a closed local port refuses immediately.

Versioning / rollout

• Auth family 0.1.5 → 0.2.0 (additive feature; the four packages version together). Publish all 4 to the Gitea feed, restore-verify.
• Zero consumer code changes — the four apps adopt by config alone at ZB deploy time. Bumping the apps' pins to 0.2.0 is a follow-up family-alignment task, not a prerequisite.
• Dev/GLAuth stacks are unaffected (no FallbackServers configured).