Introduction

Nginx is a powerful web server and reverse proxy that is used to serve many of the most popular sites in the world. In this guide, we'll demonstrate how to upgrade the Nginx executable in place, without losing client connections.

Prerequisites

Before beginning this guide, you should have a non-root user on your server, configured with sudo privileges. You will also need to have Nginx installed.

If you are using Ubuntu 14.04, you can learn how to set up a user with sudo rights here. You can install Nginx by following this guide.

If you are using CentOS 7, you can get set up by running through this guide for the sudo user, followed by this guide to install Nginx.

How the Upgrade Works

Nginx works by spawning a master process when the service starts. The master service, in turn, kicks on one or more worker processes that handle the actual client connections. Nginx is designed to perform certain actions when it receives specific signals from the administrator. Using these signals provides you with the opportunity to easily upgrade Nginx or its configuration in-place, without losing client connections.

Certain service scripts provided by distribution package maintainers will provide this ability for use with conventional upgrades. However, manually upgrading provides more flexibility in the approach and allows you to audit the upgrade to revert quickly if there are problems. This will also provide an option to upgrade gracefully if you have installed Nginx from source or using a method that does not provide this capability.

The following signals will be used:

• USR2: This spawns a new set of master/worker processes without affecting the old set.
• WINCH: This tells the Nginx master process to gracefully stop its associated worker instances.
• HUP: This tells an Nginx master process to re-read its configuration files and replace worker processes with those adhering to the new configuration. If an old and new master are running, sending this to the old master will spawn workers using their original configuration.
• QUIT: This shuts down a master and its workers gracefully.
• TERM: This initiates a fast shutdown of the master and its workers.
• KILL: This immediately kills a master and its workers without any cleanup.

Finding Nginx Process PIDs

In order to send signals to the various processes, we need to know the PID for the target process. There are two easy ways to find this.

First, you can use the ps utility and then grep for Nginx among the results. This is simple and allows you to see the master and worker processes:

ps aux | grep nginx

output
root     10846  0.0  0.3  47564  3280 ?        S    13:26   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    10847  0.0  0.1  47936  1908 ?        S    13:26   0:00 nginx: worker process
user     10961  0.0  0.0 112640   964 pts/0    S+   13:53   0:00 grep --color=auto nginx

The second column contains the PIDs for the selected processes. Highlighted is the PID for the process. The last column clarifies that the first result is an Nginx master process.

Another way to find the PID for the master Nginx process is to print out the contents of the /run/nginx.pid file:

cat /run/nginx.pid

output
10846

If there are two Nginx master processes running, the old one will be moved to /run/nginx.pid.oldbin.

Spawn a New Nginx Master/Workers Set

The first step to gracefully updating our executable is to actually update your binary. Do this using whatever method is appropriate for your Nginx installation, whether through a package manager or a source installation.

After the new binary is in place, you can spawn a second set of master/worker processes that use the new executable.

You can do this either by sending the USR2 signal directly to the PID number you queried (make sure to substitute the PID of your own Nginx master process here):

sudo kill -s USR2 10846

Or, you can read and substitute the value stored in your PID file directly into the command, like this:

sudo kill -s USR2 `cat /run/nginx.pid`

If you check your current processes, you will see that you now have two sets of Nginx master/workers:

ps aux | grep nginx

output
root     10846  0.0  0.3  47564  3280 ?        S    13:26   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    10847  0.0  0.1  47936  1908 ?        S    13:26   0:00 nginx: worker process
root     11003  0.0  0.3  47564  3132 ?        S    13:56   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    11004  0.0  0.1  47936  1912 ?        S    13:56   0:00 nginx: worker process
user     11031  0.0  0.0 112640   960 pts/0    S+   14:01   0:00 grep --color=auto nginx

You can also see that the original /run/nginx.pid file has been moved to /run/nginx.pid.oldbin and the newer master process's PID has been written to /run/nginx.pid:

tail -n +1 /run/nginx.pid*

output
==> /run/nginx.pid <==
11003

==> /run/nginx.pid.oldbin <==
10846

You can now send signals to either of the master processes using the PIDs contained in these files.

At this point, both master/worker sets are operational and capable of serving client requests. The first set is using the original Nginx executable and configuration and the second set is using the newer versions. They can continue to operate side-by-side, but for consistency, we should start to transition to the new set.

Shut Down the First Master's Workers

In order to begin the transition to the new set, the first thing we can do is stop the original master's worker processes. The original workers will finish up handling all of their current connections and then exit.

Stop the original set's workers by issuing the WINCH signal to their master process:

sudo kill -s WINCH `cat /run/nginx.pid.oldbin`

This will let the new master's workers handle new client connections alone. The old master process will still be running, but with no workers:

ps aux | grep nginx

output
root     10846  0.0  0.3  47564  3280 ?        S    13:26   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
root     11003  0.0  0.3  47564  3132 ?        S    13:56   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    11004  0.0  0.1  47936  1912 ?        S    13:56   0:00 nginx: worker process
user     11089  0.0  0.0 112640   964 pts/0    R+   14:13   0:00 grep --color=auto nginx

This lets you audit the new workers as they accept connections in isolation while maintaining the ability to revert to the old executable if something isn't quite right.

Evaluate the Outcome and Take the Next Steps

You should test and audit your system at this point to make sure that there are no signs of problems. You can leave your configuration in this state for as long as you wish to ensure that the new Nginx executable is bug-free and able to handle your traffic.

Your next step will depend entirely on whether you ran into problems.

If Your Upgrade was Successful, Complete the Transition

If you experienced no issues with your new set's workers, you can safely shut down the old master process. To do this, just send the old master the QUIT signal:

sudo kill -s QUIT `cat /run/nginx.pid.oldbin`

The old master process will exit gracefully, leaving only your new set of Nginx master/workers. At this point, you've successfully performed an in-place binary update of Nginx without interrupting client connections.

If the New Workers Experience Problems, Revert to the Old Binary

If your new set of workers seem to be having problems, you can transition back to the old configuration and binary. This is possible during the same session.

The best way to do this is to restart your old master's workers by sending it the HUP signal. Usually, when you send an Nginx master the HUP signal, it will re-read its configuration files and start new workers. However, when the target is an older master, it will just spawn new workers using its original, working configuration:

sudo kill -s HUP `cat /run/nginx.pid.oldbin`

You now should be back to having two sets of master/worker processes:

ps aux | grep nginx

output
root     10846  0.0  0.3  47564  3280 ?        S    13:26   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
root     11003  0.0  0.3  47564  3132 ?        S    13:56   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    11004  0.0  0.1  47936  1912 ?        S    13:56   0:00 nginx: worker process
nginx    19918  0.0  0.1  47936  1900 ?        S    14:47   0:00 nginx: worker process
user     19920  0.0  0.0 112640   964 pts/0    R+   14:48   0:00 grep --color=auto nginx

The newest workers are associated with the old worker. Both worker sets will be accepting client connections at this point. Now, stop the newer, buggy master process and its workers by sending the QUIT signal:

sudo kill -s QUIT `cat /run/nginx.pid`

You should be back to your old master and workers:

ps aux | grep nginx

output
root     10846  0.0  0.3  47564  3280 ?        S    13:26   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    19918  0.0  0.1  47936  1900 ?        S    14:47   0:00 nginx: worker process
user     19935  0.0  0.0 112640   964 pts/0    R+   14:50   0:00 grep --color=auto nginx

The original master will regain the /run/nginx.pid file for its PID.

If the above does not work for any reason, you can try just sending the new master server the TERM signal, which should initiate a shutdown. This should stop the new master and any workers while automatically kicking over the old master to start its worker processes. If there are serious problems and the buggy workers are not exiting, you can send each of them a KILL signal to clean up. This should be viewed as a last resort, however, as it will cut off connections.

After transitioning back to the old binary, remember that you still have the new version installed on your system. You should remove the buggy version and roll back to your previous version so that Nginx will run without issues on reboot.

Conclusion

By now, you should be able to seamlessly transition your machines from one Nginx binary to another. Nginx's ability to handle two master/workers sets while maintaining information about their relationships provides us with the ability to upgrade server software without taking the server machines offline.