LDAP Server Redundancy

When we say “zero downtime” for LDAP, we mean it. There is no excuse to have any LDAP downtime as all major server-side software vendors and open source projects have reliable replication, and all modern LDAP clients also have reliable failover, built in. And no, operator error is not an excuse for an outage. There are ways to prevent that, which we will cover in a future article.

A few things to know about keeping your LDAP system reliable and healthy:

1. Three is the right number of provider (“read-write” or “master”) servers, ideally geographically dispersed. You should send all direct writes to just one of them when business operations are as usual. You don’t get anything for load balancing writes(*), as a write to one is a replicated write to another, but it may cost you replication conflicts if you do split writes. Why three? Because if one is down, then you still have two. You can lose one, or take one down for maintenance, and still have a redundant pair.
2. If you have clients around the world, you might put your primary provider in Oakland, your secondary in Singapore, and your tertiary in Milan. Unless you have a huge volume of writes, a simple mesh topology between the providers makes most sense. Failover, when necessary, would mostly be manual or semi-automatic, i.e. someone flips the write target in the load balancer, DNS, or client config. Having a prepared single command to run to do this results in smoother outage handling.
3. LDAP is, for most practical purposes, infinitely horizontally scalable. If you see an increase in response time at your read clients’ layer, you should simply be able to add consumer replicas. Consumers should refer or chain writes to the provider (updateref / chaining), or apps will get errors when a write lands on a replica. Also, the usual cause of slow reads is a missing index or unindexed filter, not capacity. Check that before adding replicas. Size for N+1. Losing one consumer should not mean overload the rest. Design with enough headroom available to survive a traffic burst or a lost consumer at peak.
4. Load balancing can be achieved a number of ways, from DNS round robin to hardware devices like F5 to anycast. And do not forget the simplest option: a space-separated URI list in the client config gives client-side failover with no infrastructure.
5. Note for load balancer health checks that a Layer 4 (TCP) health check is not sufficient. Health checks should check LDAP health, e.g. bind result, search result (Layer 7). The check should also fail a replica whose lag exceeds a threshold, so it gets pulled from rotation. LDAP proxies like lloadd, slapd back-ldap/back-meta and PingDirectoryProxy are other alternatives.
6. Client resiliency depends on how well the software was coded and if it is configured properly.
   1. Multiple timeout settings exist, e.g. one for connection open, another for responses, another for idle time disconnects.
   2. The correct timeouts (as low as possible, but not too low) will achieve a quick failover when a server connection dies.
   3. A client with an infinite timeout setting will often hang if the server-side connection is not closed properly.
   4. If clients are connected to the server by LAN, typically the normal connection time and result times are in the single digit milliseconds, so 1 to 5 seconds might be appropriate.
   5. Writes should generally all be directed towards one provider, so a global system should have higher timeouts for clients that write to providers across a distance.
   6. Load balancer idle timeouts silently drop long-lived connections; the client will hang on its next query. TCP keepalive set below the LB idle timeout fixes it.
   7. Result timeout should allow for legitimately slow operations; one second is right for a bind or indexed lookup, but not for clients that run complex or large paged searches.
   8. Server-side idle timeouts can help quite a bit, as they reclaim memory, connection slots and file descriptors used by idle clients. Clients must handle this the same as 6(f).
7. Replication lag (contextCSN delta in OpenLDAP, RUV comparison in 389DS/RHDS) is the health signal for the whole design. Monitor and alert on it.

* There are exceptions to this. If most or all of your Singapore clients’ writes are to data (or OUs) that is only used in Singapore, by all means, let your clients in Singapore write to the providers there.