A load balancer is hardware or software that sits in front of a group of servers and spreads incoming traffic across them. Clients connect to one address; the load balancer decides which server behind it actually handles each request.
It is the piece that makes horizontal scaling usable. Adding a second application server does nothing on its own — something has to send half the traffic there. That something is the load balancer.
How it decides where to send traffic
Common strategies, roughly in order of how often you’ll see them:
- Round robin. Each request goes to the next server in the list. Simple, and fine when requests are all about the same size.
- Least connections. Send the request to whichever server is currently handling the fewest. Better when request durations vary a lot — which is exactly the case for analytics, where one query returns in 20ms and the next runs for two minutes.
- Session affinity (“sticky sessions”). Keep a given user pinned to the same server for the length of their session. Needed when servers hold state locally, and a good sign you should move that state out.
- Weighted. Give bigger servers a larger share, useful during a rolling upgrade or when your fleet isn’t uniform.
Health checks and failover
The other half of a load balancer’s job is knowing which servers are actually working. It periodically pings a health endpoint on each one; if a server stops responding, it’s pulled out of rotation until it recovers. That’s what turns a fleet of individually unreliable machines into a service that stays up, and it’s why a rolling deploy doesn’t drop requests — instances are drained one at a time rather than all restarting at once.
This is also where a lot of confusing outages come from. A health check that only verifies “the process is running” will happily keep sending traffic to a server whose database connection pool is exhausted.
In an analytics stack
You’ll run into load balancers in two places.
In front of your BI application, when you run more than one instance of it for capacity or availability. Because analytics requests are long and uneven, least-connections beats round robin here, and timeouts need to be generous enough to survive a slow query — a 60-second idle timeout will cut off exactly the reports people complain about.
And in front of the database, where a database proxy or a set of read replicas behind a single endpoint spreads read traffic while writes still go to the primary.
Either way, the load balancer is one of the better places to collect metrics: it sees every request, so its logs give you traffic volume, latency percentiles, and error rates per backend without instrumenting anything.
Related terms
Further reading
Put it to work
- Observability analytics — Overview
- Service availability dashboard — Dashboard
- Error rate — Metric
