Important SSL Security Vulnerability

On Monday, April 7th 2014, an OpenSSL vulnerability was disclosed which has been called one of the worst security holes in recent internet history. The bug, called the Heartbleed bug, was introduced in OpenSSL version 1.0.1. It has been in the wild since March of 2012 and is patched with OpenSSL version 1.0.1g released on April 7th 2014. The problem, tagged CVE-2014-0160, is described in detail here.

The bug allows any attacker to read the memory of a vulnerable host, which means that any keys that have been used on a host with a vulnerable version of OpenSSL should be considered compromised. Distributions have been updating their packages and pushing out updates, but users need to pull down the most recent packages and revoke any previous keys based on insecure versions.

We'll show you how to update your systems with a secure version of OpenSSL, revoke any insecure SSL certificates, and test whether you are vulnerable or not.

Update Your System

The DigitalOcean mirrors are being updated to include the newest versions of the OpenSSL packages as they are made available by distribution packagers. The easiest way to update your packages is to update your entire system.

On Ubuntu and Debian, you can update by typing:

sudo apt-get update
sudo apt-get dist-upgrade

If you only want to upgrade the affected packages, and not update the entire system (only recommended if you have reason to believe that upgrades to other components will break your system), you can selectively upgrade the OpenSSL packages by typing:

sudo apt-get install --only-upgrade openssl
sudo apt-get install --only-upgrade libssl1.0.0

This will upgrade the vulnerable packages while leaving the rest of your system in an un-upgraded state.

On CentOS and Fedora, you can type this to update the entire system:

yum update

If you wish to only upgrade the affected package, you can instead issue this command:

yum update openssl

Again, this is only recommended if you have a specific reason for not updating the complete system.

At the time of this writing (04/08/14), Fedora 19 does not have the latest versions in the stable repositories yet. You can wait for the update to be accepted, but you can also go ahead and manually build the package. This is probably worthwhile considering the severity of the bug.

For a 64 bit system:

yum -y install koji
koji download-build --arch=x86_64 openssl-1.0.1e-37.fc19.1
yum localinstall openssl-1.0.1e-37.fc19.1.x86_64.rpm

You can do the same for a 32-bit system of Fedora 19 by typing:

yum -y install koji
koji download-build --arch=i686 openssl-1.0.1e-37.fc19.1
yum localinstall openssl-1.0.1e-37.fc19.1.i686.rpm

On Arch Linux, the packages can be updated by typing:

sudo pacman -Syu

Arch Linux systems can become very unstable if you selectively update packages, so we do not recommend that you only update the OpenSSL package.

This should pull in all of the recent updates, including OpenSSL if the mirror you are pointed to has pulled the newest packages. You should see openssl or some variant in your list of upgraded packages.

After you are finished, you should reboot your machine to make sure your server is not using the old version loaded in memory. You can do that by issuing:

sudo shutdown -r now

Checking Your Version Numbers

You should check your version of OpenSSL after you have updated your system.

While OpenSSL version 1.0.1g is the official fix of this problem, the version that fixes this for different distributions and releases may vary. Some releases and distributions patched their older versions to fix the problem, rather than releasing an entirely new version into an older, stable ecosystem.

Because of this reason, it is best to check through your distribution's packaging system, since the openssl version command might not reflect the information we need.

Debian and Ubuntu Releases and Fix Versions

For Debian and Ubuntu systems, you get the current version of your OpenSSL package by typing:

dpkg -l | grep "openssl"

You should receive output like this:

ii  openssl                            1.0.1e-2+deb7u6               amd64        Secure Socket Layer (SSL) binary and related cryptographic tools

For Debian users, the release of Debian that you are running will determine the correct version for the fix. If your version of OpenSSL is at least as recent as the version listed here for your distribution, you should be protected:

• Debian 6 (Squeeze): Unaffected (Shipped with older version prior to vulnerability)
• Debian 7 (Wheezy): 1.0.1e-2+deb7u6
• Debian testing (Jessie): 1.0.1g-1
• Debian unstable (Sid): 1.0.1g-1

For Ubuntu users, the correct, patched version is also release-dependent. Use this list to see the minimum secure version for your release:

• Ubuntu 10.04: Unaffected (Shipped with older version prior to vulnerability)
• Ubuntu 12.04: 1.0.1-4ubuntu5.12
• Ubuntu 12.10: 1.0.1c-3ubuntu2.7
• Ubuntu 13.04: SUPPORT END OF LIFE REACHED, SHOULD UPGRADE
• Ubuntu 13.10: 1.0.1e-3ubuntu1.2

If you are on one of the supported distros, ensure that your OpenSSL version is up-to-date. If your distribution is not supported anymore (Ubuntu 13.04), it is highly recommended that you transition to a supported operating system due to this bug's severity.

CentOS and Fedora Releases and Fix Versions

For CentOS and Fedora systems, you can query the version of the OpenSSL package installed on your system by typing:

rpm -q -a | grep "openssl"

You should receive output that looks like this:

openssl-1.0.1e-16.el6_5.7.x86_64

For CentOS, here are the releases and the minimum versions of OpenSSL that must be applied to protect future SSL interactions. We will take the architecture off the end in our list:

• CentOS 5: Unaffected (Shipped with older version prior to vulnerability)
• CentOS 6: openssl-1.0.1e-16.el6.5.7

For Fedora users, you can check that your package version is at least as recent as the ones listed below. Again, I have removed the architecture below because this applies to both 32-bit and 64-bit releases:

• Fedora 17: Unaffected (Shipped with older version prior to vulnerability)
• Fedora 19: openssl-1.0.1e-37.fc19.1

Note: Pay close attention to the Fedora version number. The trailing ".1" tells you if it is patched or not. If your package does not have the ".1" at the end, you are still vulnerable!

Arch Linux Fix Versions

If you are running Arch Linux, the validation is much easier since there are not multiple releases.

You can check the version of OpenSSL that you have installed with:

pacman -Q | grep "openssl"

You should receive output that looks like this:

openssl 1.0.1.g-1

If your installed version is this release or higher, you are okay.

Revoking and Reissuing Your SSL Certs/Keys

If you have purchased an SSL certificate from a provider and you have updated your OpenSSL packages on your server, you will need to revoke your old keys and you'll have to reissue new keys. This is a process known as "rekeying".

This process is very dependent upon the SSL service that issued your initial certificate, but you should search their administration interface for an option that is similar to "rekey" or "reissue keys". Most SSL issuers will revoke your former key when you rekey, but you can usually also do this explicitly using their administrative interface.

Follow the directions that your SSL provider gives you. They may give you very specific instructions for how to regenerate a CSR, or they may not.

If they do not provide you with the specific openssl commands that they would like you to use, you can generate your new SSL CSR by typing something like this. Again, add sudo if you are not root:

openssl req -new -newkey rsa:2048 -nodes -keyout hostname.key -out hostname.csr

You will need to copy your generated CSR into your provider's web interface after generation in order to rekey your server. You will then need to download the new certificate from the web interface.

You will have to install the new keys to the same location that your old keys and certificates were kept. The path that you'll need to use for your certificate and keys will vary by distribution and how you configured your web server. For instance, some are kept in /etc/ssl/certs while others may be kept in locations provided by your web server.

For example, if you are using the Apache web server, you should see a line in your main Apache configuration file, a virtualhost file, or a separately sourced configuration file that points to the location where it looks for your SSL information:

SSLEngine on
SSLCertificateFile /path/to/your_domain.crt
SSLCertificateKeyFile /path/to/your_key.key
SSLCertificateChainFile /path/to/CA.crt

These may look different, but they should point you in the correct direction to find your SSL certificate location.

If you are using Nginx, you'll find similar directives that point to your server's SSL certificate and key. They might look something like this:

server {
    . . .
    ssl_certificate /path/to/your_domain.crt;
    ssl_certificate_key /path/to/your_key.key;
    . . .
}

Another option is to check your distribution's documentation or check your server's filesystem to find out where your certificates are stored.

When you are finished, you should restart your web server to use the new keys. The method of doing this will vary by distribution and server.

On Debian or Ubuntu, you can restart your web server by typing:

sudo service apache2 restart    # For Apache web server
sudo service nginx restart      # For Nginx web server

On CentOS or Fedora, you can restart by typing:

sudo service httpd restart      # For Apache web server
sudo service nginx restart      # For Nginx web server

For Arch Linux you should use something like:

sudo systemctl restart httpd.service
sudo systemctl restart nginx.service

This should allow your web server to pick up your new certificate changes.

Additional Considerations from a Client's Perspective

Because of the widespread nature of this bug, there are other considerations that you should take into account as well. As a consumer of web services and sites, you should also react quickly to try to minimize the potential damage to your accounts and information.

You should consider any communication that you secured by SSL previously to have been compromised by this bug. This means any kind of interaction with secure websites were open to snooping.

A good first step is to change your password on every site that you use, after you have verified that they have updated their OpenSSL versions to patch this vulnerability. If you change your password prior to the remote site patching their SSL version, your new password is just as vulnerable as your old one.

One consideration that is of high importance is to secure any VPN instances that you have set up. There are a few different ways that VPN connections are implemented, but SSL is one of the most popular. For instance, OpenVPN uses SSL. Any certificates required to connect to your server should be regenerated to ensure that they are secured.

Another good measure is to remove all session keys and cookies. This means regenerate API keys, clear cookies that are stored in your browser, etc. This may be a massive inconvenience, but the cost of not going through these pains now is that your accounts are basically wide open and communication with remote servers that have not updated their OpenSSL should be considered no more secure than plain-text.

Conclusion

This is an incredibly large bug that users and administrators cannot afford to ignore or put off. You should update any servers that you have control over immediately and inform your users of the vulnerabilities so that they can do damage control from their end.

By patching your system and updating your SSL certificates, you should be protected against this vulnerability, as a host, for future communications.