Problem scenario
To improve performance of your HTTP load balancing mechanism, you want to configure how failed members of the load balancer are dealt with. Some Nginx instances may fail in your environment. You do not want attempts to send web traffic to these instances immediately after several failures. How do you granularly keep track of failed attempts and remove the instance from the distribution algorithm after a certain limit has been reach to improve performance by attempting only more reliable instances?
Solution
Prerequisite
You must have configured Nginx as a an HTTP load balancer (or reverse proxy server). To do this, see this article which will actually work for Nginx distributing traffic to regular Nginx websites, Apache websites, or Nginx websites running in Docker containers.
Procedures
Use Nginx's built-in "max_fails" and "fail_timeout" directives in the default.conf file. These reserved words can allow you to easily configure a threshold of failures and a duration that the server should not be tried again. Depending on your environment and how quickly the Nginx server usually returns to being operational, you can adjust accordingly. Here is an example snippet from the default.conf file:
upstream backend {
server 10.10.10.10 max_fails=5 fail_timeout=20s;
server 10.10.10.11 max_fails=5 fail_timeout=20s;
server 10.10.10.12 max_fails=10 fail_timeout=5s;
server 10.10.10.13 max_fails=2 fail_timeout=100s;
}
These max_fails and fail_timeout settings work with Nginx in Docker no differently from Nginx running on a server directly. The default timeout is 10 seconds (according to DigitalOcean's website).