scadaproj/docs/plans/2026-08-10-ldap-backup-dc-failover-design.md

# LDAP backup-DC failover — design (`ZB.MOM.WW.Auth` 0.2.0)

**Date:** 2026-08-10
**Status:** Approved
**Motivation:** Production servers at ZB authenticate against the Zimmer NAM AD
([`ldap_setup.md`](../../ldap_setup.md)), which has two domain controllers
(`uswap-vdc-nam1` / `uswap-vdc-nam2`). `ZB.MOM.WW.Auth.Ldap` takes a single
`Server`, so a primary-DC outage blocks every interactive login family-wide
(API keys unaffected). The working Ignition gateway has primary + secondary DC
failover; this brings the shared library to parity on that axis.

## Decisions (made with the user, 2026-08-10)

1. **Sticky preference** failover — try the last-known-good DC first; on failure
   walk the ordered list and remember whichever worked. Mirrors the Secrets
   0.5.0 `FallbackEndpoints` sticky/no-flapping precedent. Rejected:
   always-primary-first (pays the full connection timeout on every login during
   a primary outage), sticky + probe-back (YAGNI for two DCs).
2. **Failover-only** — no Ignition-style credential cache. A both-DCs-down event
   is rare and correlated with wider outages; API keys already survive it; a
   credential cache adds credential-at-rest surface. Possible future follow-up.
3. **Approach A: service-level loop** — the walk lives in `LdapAuthService`,
   which already owns failure classification (user-fault vs system-fault).
   Rejected: connection-factory-level (only sees `Connect`; a DC that accepts
   TCP+TLS but fails service-bind/search would never fail over), DNS-level
   `nam.zimmer.com` (DC certs' SANs carry only their own FQDN → LDAPS hostname
   validation breaks; no stickiness or timeout control).

## Probe evidence (2026-08-10, recorded in `ldap_setup.md`)

Both DCs serve LDAPS (636, TLSv1.3) and StartTLS (389); anonymous rootDSE over
LDAPS returns `result: 0 Success` on both. Certs: per-host FQDN SANs, issued by
`zimmer-issueca1` ← `zimmer-rootca` (internal CA, root not served), valid to
2027-03. **Caveat: both certs are RSA 1024-bit** — OpenSSL at SECLEVEL=2
(Debian-based .NET containers) rejects the handshake (`EE certificate key too
weak`); Windows/Schannel hosts accept. Container deployments need a lowered
SECLEVEL, a validation-callback path, or (right fix) a reissued 2048-bit DC
cert template.

## Config surface

One additive field on `LdapOptions` (`ZB.MOM.WW.Auth.Abstractions`):

```csharp
/// <summary>Ordered fallback LDAP endpoints, "host" or "host:port"; a bare host uses <see cref="Port"/>.</summary>
public IReadOnlyList<string> FallbackServers { get; init; } = Array.Empty<string>();
```

```jsonc
"Ldap": {
  "Server": "uswap-vdc-nam1.nam.zimmer.com",
  "Port": 636,
  "Transport": "Ldaps",
  "FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com" ]
}
```

- Empty (the default) ⇒ behavior byte-identical to today: single endpoint, no
  loop, no new state.
- `ServerCertificateValidationCallback` and `AllowInsecure` apply to every
  endpoint (one CA family — verified by the probe).
- **Validator** (`LdapOptionsValidator`, only when `Enabled`): each entry must
  have a non-empty host; an explicit port must parse as 1–65535. Malformed
  entries fail boot with a field-naming message in the validator's existing
  style. Duplicates are permitted (harmless; de-duplication is not the
  validator's job).

## Failover semantics

Inside `LdapAuthService.Authenticate`, an endpoint walk over an internal
`LdapEndpoint(string Host, int Port)` list = `Server:Port` + parsed
`FallbackServers`, in declared order:

- **Walk order:** sticky-preferred endpoint first, then the remaining endpoints
  in declared order (skipping the preferred). On success at endpoint *i*,
  preference := *i*.
- **Only system-side failures advance the walk:** connect/TLS-upgrade failure,
  service-account bind failure, search failure, and any unexpected exception
  inside an attempt. **User-side results return immediately, never fail over:**
  `UserNotFound`, `AmbiguousUser`, `BadCredentials` (user bind),
  `GroupLookupFailed` — those are answers from a healthy directory.
- **Exhaustion:** all endpoints failed system-side ⇒ return the last failure
  (maps to `ServiceAccountBindFailed`, as today — no `LdapAuthFailure` enum
  change, no `Abstractions` contract change beyond the new option).
- **Fail-closed contract preserved:** `AuthenticateAsync` still never throws;
  each per-endpoint attempt is individually guarded.
- Documented trade-offs:
  - A genuinely wrong service-account password tries every DC per login (fails
    fast at bind — no timeout cost; 2 endpoints in practice).
  - Worst-case login latency while the preferred DC is down =
    `ConnectionTimeoutMs` × walk position, paid once — stickiness then moves
    preference to the survivor.
  - Sticky means the primary stays unused after recovery until the current
    preferred fails; when it does, the walk tries declared order again, so
    preference drifts back to the primary naturally.

> **Refinement (Task 4):** sticky preference updates on ANY authoritative answer
> (success or user-side failure), not only on success — a DC that answers is healthy,
> and success-only stickiness would re-pay the dead primary's connect timeout on every
> mistyped password during an outage. Corollary, also implemented: an *exhausted* walk
> (every endpoint system-side-down) leaves the preference untouched, since a total
> outage says nothing about which endpoint to prefer once one recovers.

## Stickiness state

`volatile int _preferredIndex` on the singleton `LdapAuthService` (registered
`TryAddSingleton` in `ZB.MOM.WW.Auth.AspNetCore`). Benign races tolerated —
worst case is one redundant failed attempt. No cross-process or persisted
state.

## Observability

New optional `ILogger<LdapAuthService>?` constructor parameter (null default —
existing constructors keep compiling; adds only
`Microsoft.Extensions.Logging.Abstractions` to `ZB.MOM.WW.Auth.Ldap`). The
AspNetCore registration passes the host logger. Log points:

- **Warning** per failover hop: endpoint, stage that failed, next endpoint.
- **Information** on sticky-preference change (old → new endpoint).
- Nothing on the happy path; never log credentials or passwords.

Rationale: silent failover hides a dead primary DC indefinitely — the feature
working correctly must be visible to operators.

## Testing

- **Unit (FakeLdapConnection seam):** failover triggered by each system-side
  stage (connect, service bind, search, unexpected exception); NO failover on
  each user-side result; stickiness updates on success and drifts back after
  the preferred fails; exhaustion returns `ServiceAccountBindFailed`; empty
  `FallbackServers` short-circuits to today's exact behavior; endpoint parsing
  (`host`, `host:port`, malformed).
- **Validator tests:** new cases in `LdapOptionsValidatorTests` style.
- **Integration (env-gated GLAuth suite):** primary pointed at a closed local
  port + fallback at the real GLAuth ⇒ login succeeds, preference sticks, and a
  second login does not retry the dead primary. Fast offline too — a closed
  local port refuses immediately.

## Versioning / rollout

- Auth family `0.1.5` → **`0.2.0`** (additive feature; the four packages
  version together). Publish all 4 to the Gitea feed, restore-verify.
- **Zero consumer code changes** — the four apps adopt by config alone at ZB
  deploy time. Bumping the apps' pins to 0.2.0 is a follow-up family-alignment
  task, not a prerequisite.
- Dev/GLAuth stacks are unaffected (no `FallbackServers` configured).