How Load Balancers Work

Hatchbox uses Caddy for serving applications as well as load balancing. When enabled, Caddy is configured to load balance requests to upstream web servers with retries and health checks automatically enabled.

Load Balancing

By default, a "least connections" policy is used for routing traffic to upstream web servers. This distributes traffic to the least busy servers first.

You can customize the load balancing policy in the Cluster's settings page to use any of the available load balancing policies.

Passive Health Checks

Load balancers are configured with passive health checks by default. When a request is forwarded to a web server, the request is monitored for failures which is used to determine the upstream's health state.

• Failed requests will be remembered for 30 seconds.
• 2 failures will mark an upstream as unhealthy.
• Responses with a status of 502, 503, or 504 are considered unhealthy.

Active Health Checks

We highly recommend enabling active health checks in your App's settings.

Active health checks make a request from the load balancer to each upstream on a schedule to check their health. If the upstream does not return a 200 OK response, it is considered unhealthy and removed from the load balancer's rotation until it is marked healthy again.

To ensure that health checks are routed to the correct application in a cluster, the Host header is set to your application's hatchboxapp.com subdomain.

If your application enables host authorization (like Rails Host Authorization), make sure to exclude your health check endpoint.

  # config/environments/production.rb in Rails
  
  # Enable DNS rebinding protection and other `Host` header attacks.
config.hosts = [
  "example.com",     # Allow requests from example.com
  /.*\.example\.com/ # Allow requests from subdomains like `www.example.com`
  ]
  
  # Skip DNS rebinding protection for the default health check endpoint.
  config.host_authorization = { exclude: ->(request) { request.path == "/up" } }

Retries

When a request to an upstream fails, the load balancer is configured to retry requests multiple times. This only applies to GET requests since they are safe to retry.

By default, requests will be retried for up to 5 seconds.