DigitalOcean Community Tutorials

An Introduction to the Kubernetes DNS Service

2018-10-17T20:22:37Z

Introduction

The Domain Name System (DNS) is a system for associating various types of information – such as IP addresses – with easy-to-remember names. By default most Kubernetes clusters automatically configure an internal DNS service to provide a lightweight mechanism for service discovery. Built-in service discovery makes it easier for applications to find and communicate with each other on Kubernetes clusters, even when pods and services are being created, deleted, and shifted between nodes.

The implementation details of the Kubernetes DNS service have changed in recent versions of Kubernetes. In this article we will take a look at both the kube-dns and CoreDNS versions of the Kubernetes DNS service. We will review how they operate and the DNS records that Kubernetes generates.

To gain a more thorough understanding of DNS before you begin, please read An Introduction to DNS Terminology, Components, and Concepts. For any Kubernetes topics you may be unfamiliar with, you could read An Introduction to Kubernetes.

What Does the Kubernetes DNS Service Provide?

Before Kubernetes version 1.11, the Kubernetes DNS service was based on kube-dns. Version 1.11 introduced CoreDNS to address some security and stability concerns with kube-dns.

Regardless of the software handling the actual DNS records, both implementations work in a similar manner:

A service named kube-dns and one or more pods are created.
The kube-dns service listens for service and endpoint events from the Kubernetes API and updates its DNS records as needed. These events are triggered when you create, update or delete Kubernetes services and their associated pods.
kubelet sets each new pod's /etc/resolv.conf nameserver option to the cluster IP of the kube-dns service, with appropriate search options to allow for shorter hostnames to be used:

resolv.conf
```
nameserver 10.32.0.10
search namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```
Applications running in containers can then resolve hostnames such as example-service.namespace into the correct cluster IP addresses.

Example Kubernetes DNS Records

The full DNS A record of a Kubernetes service will look like the following example:

service.namespace.svc.cluster.local

A pod would have a record in this format, reflecting the actual IP address of the pod:

10.32.0.125.namespace.pod.cluster.local

Additionally, SRV records are created for a Kubernetes service's named ports:

_port-name._protocol.service.namespace.svc.cluster.local

The result of all this is a built-in, DNS-based service discovery mechanism, where your application or microservice can target a simple and consistent hostname to access other services or pods on the cluster.

Search Domains and Resolving Shorter Hostnames

Because of the search domain suffixes listed in the resolv.conf file, you often won't need to use the full hostname to contact another service. If you're addressing a service in the same namespace, you can use just the service name to contact it:

other-service

If the service is in a different namespace, add it to the query:

other-service.other-namespace

If you're targeting a pod, you'll need to use at least the following:

pod-ip.other-namespace.pod

As we saw in the default resolv.conf file, only .svc suffixes are automatically completed, so make sure you specify everything up to .pod.

Now that we know the practical uses of the Kubernetes DNS service, let's run through some details on the two different implementations.

Kubernetes DNS Implementation Details

As noted in the previous section, Kubernetes version 1.11 introduced new software to handle the kube-dns service. The motivation for the change was to increase the performance and security of the service. Let's take a look at the original kube-dns implementation first.

kube-dns

The kube-dns service prior to Kubernetes 1.11 is made up of three containers running in a kube-dns pod in the kube-system namespace. The three containers are:

kube-dns: a container that runs SkyDNS, which performs DNS query resolution
dnsmasq: a popular lightweight DNS resolver and cache that caches the responses from SkyDNS
sidecar: a sidecar container that handles metrics reporting and responds to health checks for the service

Security vulnerabilities in Dnsmasq, and scaling performance issues with SkyDNS led to the creation of a replacement system, CoreDNS.

CoreDNS

As of Kubernetes 1.11 a new Kubernetes DNS service, CoreDNS has been promoted to General Availability. This means that it's ready for production use and will be the default cluster DNS service for many installation tools and managed Kubernetes providers.

CoreDNS is a single process, written in Go, that covers all of the functionality of the previous system. A single container resolves and caches DNS queries, responds to health checks, and provides metrics.

In addition to addressing performance- and security-related issues, CoreDNS fixes some other minor bugs and adds some new features:

Some issues with incompatibilities between using stubDomains and external services have been fixed
CoreDNS can enhance DNS-based round-robin load balancing by randomizing the order in which it returns certain records
A feature called autopath can improve DNS response times when resolving external hostnames, by being smarter about iterating through each of the search domain suffixes listed in resolv.conf
With kube-dns 10.32.0.125.namespace.pod.cluster.local would always resolve to 10.32.0.125, even if the pod doesn't actually exist. CoreDNS has a "pods verified" mode that will only resolve successfully if a pod exists with the right IP and in the right namespace.

For more information on CoreDNS and how it differs from kube-dns, you can read the Kubernetes CoreDNS GA announcement.

Additional Configuration Options

Kubernetes operators often want to customize how their pods and containers resolve certain custom domains, or need to adjust the upstream nameservers or search domain suffixes configured in resolv.conf. You can do this with the dnsConfig option of your pod's spec:

example_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  namespace: example
  name: custom-dns
spec:
  containers:
    - name: example
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 203.0.113.44
    searches:
      - custom.dns.local

Updating this config will rewrite a pod's resolv.conf to enable the changes. The configuration maps directly to the standard resolv.conf options, so the above config would create a file with nameserver 203.0.113.44 and search custom.dns.local lines.

Conclusion

In this article we covered the basics of what the Kubernetes DNS service provides to developers, showed some example DNS records for services and pods, discussed how the system is implemented on different Kubernetes versions, and highlighted some additional configuration options available to customize how your pods resolve DNS queries.

For more information on the Kubernetes DNS service, please refer to the official Kubernetes DNS for Services and Pods documentation.

How to Install and Configure pgAdmin 4 in Server Mode

2018-10-19T14:21:46Z

Introduction

pgAdmin is an open-source administration and development platform for PostgreSQL and its related database management systems. Written in Python and jQuery, it supports all the features found in PostgreSQL. You can use pgAdmin to do everything from writing basic SQL queries to monitoring your databases and configuring advanced database architectures.

In this tutorial, we'll walk through the process of installing and configuring the latest version of pgAdmin onto an Ubuntu 18.04 server, accessing pgAdmin through a web browser, and connecting it to a PostgreSQL database on your server.

Prerequisites

To complete this tutorial, you will need:

A server running Ubuntu 18.04. This server should have a non-root user with sudo privileges, as well as a firewall configured with ufw. For help with setting this up, follow our Initial Server Setup Guide for Ubuntu 18.04.
The Apache web server installed on your server. Follow our guide on How To Install the Apache Web Server on Ubuntu 18.04 to configure this on your machine.
PostgreSQL installed on your server. You can set this up by following our guide on How To Install and Use PostgreSQL on Ubuntu 18.04. As you follow this guide, be sure to create a new role and database, as you will need both to connect pgAdmin to your PostgreSQL instance.
Python 3 and venv installed on your server. Follow How To Install Python 3 and Set Up a Programming Environment on an Ubuntu 18.04 server to install these tools and set up a virtual environment.

Step 1 — Installing pgAdmin and its Dependencies

As of this writing, the most recent version of pgAdmin is pgAdmin 4, while the most recent version available through the official Ubuntu repositories is pgAdmin 3. pgAdmin 3 is no longer supported though, and the project maintainers recommend installing pgAdmin 4. In this step, we will go over the process of installing the latest version of pgAdmin 4 within a virtual environment (as recommended by the project's development team) and installing its dependencies using apt.

To begin, update your server’s package index if you haven't done so recently:

sudo apt update

Next, install the following dependencies. These include libgmp3-dev, a multiprecision arithmetic library; libpq-dev, which includes header files and a static library that helps communication with a PostgreSQL backend; and libapache2-mod-wsgi-py3, an Apache module that allows you to host Python-based web applications within Apache:

sudo apt install libgmp3-dev libpq-dev libapache2-mod-wsgi-py3

Following this, create a few directories where pgAdmin will store its sessions data, storage data, and logs:

sudo mkdir -p /var/lib/pgadmin4/sessions
sudo mkdir /var/lib/pgadmin4/storage
sudo mkdir /var/log/pgadmin4

Then, change ownership of these directories to your non-root user and group. This is necessary because they are currently owned by your root user, but we will install pgAdmin from a virtual environment owned by your non-root user, and the installation process involves creating some files within these directories. After the installation, however, we will change the ownership over to the www-data user and group so it can be served to the web:

sudo chown -R sammy:sammy /var/lib/pgadmin4
sudo chown -R sammy:sammy /var/log/pgadmin4

Next, open up your virtual environment. Navigate to the directory your programming environment is in and activate it. Following the naming conventions of the prerequisite Python 3 tutorial, we’ll go to the environments directory and activate the my_env environment:

cd environments/
source my_env/bin/activate

Following this, download the pgAdmin 4 source code onto your machine. To find the latest version of the source code, navigate to the pgAdmin 4 (Python Wheel) Download page and click the link for the latest version (v3.4, as of this writing). This will take you to a Downloads page on the PostgreSQL website. Once there, copy the file link that ends with .whl — the standard built-package format used for Python distributions. Then go back to your terminal and run the following wget command, making sure to replace the link with the one you copied from the PostgreSQL site, which will download the .whl file to your server:

wget https://ftp.postgresql.org/pub/pgadmin/pgadmin4/v3.4/pip/pgadmin4-3.4-py2.py3-none-any.whl

Next install the wheel package, the reference implementation of the wheel packaging standard. A Python library, this package serves as an extension for building wheels and includes a command line tool for working with .whl files:

python -m pip install wheel

Then install pgAdmin 4 package with the following command:

python -m pip install pgadmin4-3.4-py2.py3-none-any.whl

That takes care of installing pgAdmin and its dependencies. Before connecting it to your database, though, there are a few changes you’ll need to make to the program’s configuration.

Step 2 — Configuring pgAdmin 4

Although pgAdmin has been installed on your server, there are still a few steps you must go through to ensure it has the permissions and configurations needed to allow it to correctly serve the web interface.

pgAdmin’s main configuration file, config.py, is read before any other configuration file. Its contents can be used as a reference point for further configuration settings that can be specified in pgAdmin’s other config files, but to avoid unforeseen errors, you should not edit the config.py file itself. We will add some configuration changes to a new file, named config_local.py, which will be read after the primary one.

Create this file now using your preferred text editor. Here, we will use nano:

nano my_env/lib/python3.6/site-packages/pgadmin4/config_local.py

In your editor, add the following content:

environments/my_env/lib/python3.6/site-packages/pgadmin4/config_local.py

LOG_FILE = '/var/log/pgadmin4/pgadmin4.log'
SQLITE_PATH = '/var/lib/pgadmin4/pgadmin4.db'
SESSION_DB_PATH = '/var/lib/pgadmin4/sessions'
STORAGE_DIR = '/var/lib/pgadmin4/storage'
SERVER_MODE = True

Here are what these five directives do:

LOG_FILE: this defines the file in which pgAdmin’s logs will be stored.
SQLITE_PATH: pgAdmin stores user-related data in an SQLite database, and this directive points the pgAdmin software to this configuration database. Because this file is located under the persistent directory /var/lib/pgadmin4/, your user data will not be lost after you upgrade.
SESSION_DB_PATH: specifies which directory will be used to store session data.
STORAGE_DIR: defines where pgAdmin will store other data, like backups and security certificates.
SERVER_MODE: setting this directive to True tells pgAdmin to run in Server mode, as opposed to Desktop mode.

Notice that each of these file paths point to the directories you created in Step 1.

After adding these lines, save and close the file (press CTRL + X, followed by Y and then ENTER). With those configurations in place, run the pgAdmin setup script to set your login credentials:

python my_env/lib/python3.6/site-packages/pgadmin4/setup.py

After running this command, you will see a prompt asking for your email address and a password. These will serve as your login credentials when you access pgAdmin later on, so be sure to remember or take note of what you enter here:

Output. . .
Enter the email address and password to use for the initial pgAdmin user account:

Email address: sammy@example.com
Password: 
Retype password:

Following this, deactivate your virtual environment:

deactivate

Recall the file paths you specified in the config_local.py file. These files are held within the directories you created in Step 1, which are currently owned by your non-root user. They must, however, be accessible by the user and group running your web server. By default on Ubuntu 18.04, these are the www-data user and group, so update the permissions on the following directories to give www-data ownership over both of them:

sudo chown -R www-data:www-data /var/lib/pgadmin4/
sudo chown -R www-data:www-data /var/log/pgadmin4/

With that, pgAdmin is fully configured. However, the program isn't yet being served from your server, so it remains inaccessible. To resolve this, we will configure Apache to serve pgAdmin so you can access its user interface through a web browser.

Step 3 — Configuring Apache

The Apache web server uses virtual hosts to encapsulate configuration details and host more than one domain from a single server. If you followed the prerequisite Apache tutorial, you may have set up an example virtual host file under the name example.com.conf, but in this step we will create a new one from which we can serve the pgAdmin web interface.

To begin, make sure you're in your root directory:

cd /

Then create a new file in your /sites-available/ directory called pgadmin4.conf. This will be your server’s virtual host file:

sudo nano /etc/apache2/sites-available/pgadmin4.conf

Add the following content to this file, being sure to update the highlighted parts to align with your own configuration:

/etc/apache2/sites-available/pgadmin4.conf

<VirtualHost *>
    ServerName your_server_ip

    WSGIDaemonProcess pgadmin processes=1 threads=25 python-home=/home/sammy/environments/my_env
    WSGIScriptAlias / /home/sammy/environments/my_env/lib/python3.6/site-packages/pgadmin4/pgAdmin4.wsgi

    <Directory "/home/sammy/environments/my_env/lib/python3.6/site-packages/pgadmin4/">
        WSGIProcessGroup pgadmin
        WSGIApplicationGroup %{GLOBAL}
        Require all granted
    </Directory>
</VirtualHost>

Save and close the virtual host file. Next, use the a2dissite script to disable the default virtual host file, 000-default.conf:

sudo a2dissite 000-default.conf

Note: If you followed the prerequisite Apache tutorial, you may have already disabled 000-default.conf and set up an example virtual host configuration file (named example.com.conf in the prerequisite). If this is the case, you will need to disable the example.com.conf virtual host file with the following command:

sudo a2dissite example.com.conf

Then use the a2ensite script to enable your pgadmin4.conf virtual host file. This will create a symbolic link from the virtual host file in the /sites-available/ directory to the /sites-enabled/ directory:

sudo a2ensite pgadmin4.conf

Following this, test that your configuration file’s syntax is correct:

apachectl configtest

If your configuration file is all in order, you will see Syntax OK. If you see an error in the output, reopen the pgadmin4.conf file and double check that your IP address and file paths are all correct, then rerun the configtest.

Once you see Syntax OK in your output, restart the Apache service so it reads your new virtual host file:

sudo systemctl restart apache2

pgAdmin is now fully installed and configured. Next, we'll go over how to access pgAdmin from a browser before connecting it to your PostgreSQL database.

Step 4 — Accessing pgAdmin

On your local machine, open up your preferred web browser and navigate to your server’s IP address:

http://your_server_ip

Once there, you’ll be presented with a login screen similar to the following:

Enter the login credentials you defined in Step 2, and you’ll be taken to the pgAdmin Welcome Screen:

Now that you've confirmed you can access the pgAdmin interface, all that's left to do is to connect pgAdmin to your PostgreSQL database. Before doing so, though, you'll need to make one minor change to your PostgreSQL superuser's configuration.

Step 5 — Configuring your PostgreSQL User

If you followed the prerequisite PostgreSQL tutorial, you should already have PostgreSQL installed on your server with a new superuser role and database set up.

By default in PostgreSQL, you authenticate as database users using the "Identification Protocol," or "ident," authentication method. This involves PostgreSQL taking the client's Ubuntu username and using it as the allowed database username. This can allow for greater security in many cases, but it can also cause issues in instances where you'd like an outside program, such as pgAdmin, to connect to one of your databases. To resolve this, we will set a password for this PostgreSQL role which will allow pgAdmin to connect to your database.

From your terminal, open the PostgreSQL prompt under your superuser role:

sudo -u sammy psql

From the PostgreSQL prompt, update the user profile to have a strong password of your choosing:

ALTER USER sammy PASSWORD 'password';

Then exit the PostgreSQL prompt:

\q

Next, go back to the pgAdmin 4 interface in your browser, and locate the Browser menu on the left hand side. Right-click on Servers to open a context menu, hover your mouse over Create, and click Server….

This will cause a window to pop up in your browser in which you'll enter info about your server, role, and database.

In the General tab, enter the name for this server. This can be anything you'd like, but you may find it helpful to make it something descriptive. In our example, the server is named Sammy-server-1.

Next, click on the Connection tab. In the Host name/address field, enter localhost. The Port should be set to 5432 by default, which will work for this setup, as that's the default port used by PostgreSQL.

In the Maintenance database field, enter the name of the database you'd like to connect to. Note that this database must already be created on your server. Then, enter the PostgreSQL username and password you configured previously in the Username and Password fields, respectively.

The empty fields in the other tabs are optional, and it's only necessary that you fill them in if you have a specific setup in mind in which they're required. Click the Save button, and the database will appear under the Servers in the Browser menu.

You've successfully connected pgAdmin4 to your PostgreSQL database. You can do just about anything from the pgAdmin dashboard that you would from the PostgreSQL prompt. To illustrate this, we will create an example table and populate it with some sample data through the web interface.

Step 6 — Creating a Table in the pgAdmin Dashboard

From the pgAdmin dashboard, locate the Browser menu on the left-hand side of the window. Click on the plus sign (+) next to Servers (1) to expand the tree menu within it. Next, click the plus sign to the left of the server you added in the previous step (Sammy-server-1 in our example), then expand Databases, the name of the database you added (sammy, in our example), and then Schemas (1). You should see a tree menu like the following:

Right-click the Tables list item, then hover your cursor over Create and click Table….

This will open up a Create-Table window. Under the General tab of this window, enter a name for the table. This can be anything you'd like, but to keep things simple we'll refer to it as table-01.

Then navigate to the Columns tab and click the + sign in the upper right corner of the window to add some columns. When adding a column, you're required to give it a Name and a Data type, and you may need to choose a Length if it's required by the data type you've selected.

Additionally, the official PostgreSQL documentation states that adding a primary key to a table is usually best practice. A primary key is a constraint that indicates a specific column or set of columns that can be used as a special identifier for rows in the table. This isn't a requirement, but if you'd like to set one or more of your columns as the primary key, toggle the switch at the far right from No to Yes.

Click the Save button to create the table.

By this point, you've created a table and added a couple columns to it. However, the columns don't yet contain any data. To add data to your new table, right-click the name of the table in the Browser menu, hover your cursor over Scripts and click on INSERT Script.

This will open a new panel on the dashboard. At the top you'll see a partially-completed INSERT statement, with the appropriate table and column names. Go ahead and replace the question marks (?) with some dummy data, being sure that the data you add aligns with the data types you selected for each column. Note that you can also add multiple rows of data by adding each row in a new set of parentheses, with each set of parentheses separated by a comma as shown in the following example.

If you'd like, feel free to replace the partially-completed INSERT script with this example INSERT statement:

INSERT INTO public."table-01"(
    col1, col2, col3)
    VALUES ('Juneau', 14, 337), ('Bismark', 90, 2334), ('Lansing', 51, 556);

Click on the lightning bolt icon (⚡) to execute the INSERT statement. To view the table and all the data within it, right-click the name of your table in the Browser menu once again, hover your cursor over View/Edit Data, and select All Rows.

This will open another new panel, below which, in the lower panel's Data Output tab, you can view all the data held within that table.

With that, you've successfully created a table and populated it with some data through the pgAdmin web interface. Of course, this is just one method you can use to create a table through pgAdmin. For example, it's possible to create and populate a table using SQL instead of the GUI-based method described in this step.

Conclusion

In this guide, you learned how to install pgAdmin 4 from a Python virtual environment, configure it, serve it to the web with Apache, and how to connect it to a PostgreSQL database. Additionally, this guide went over one method that can be used to create and populate a table, but pgAdmin can be used for much more than just creating and editing tables.

For more information on how to get the most out of all of pgAdmin's features, we encourage you to review the project's documentation. You can also learn more about PostgreSQL through our Community tutorials on the subject.

How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image

2018-10-17T21:46:57Z

Introduction

DigitalOcean's Custom Images feature allows you to bring your virtual disk images from an on-premise environment or another cloud platform to DigitalOcean and use them to start DigitalOcean Droplets.

As described in the Custom Images documentation, the following image types are supported natively by the Custom Images upload tool:

ISO is another popular image format which you may want to use with Custom Images. ISO images are frequently provided by Linux distributions as a convenient method for installing Linux. Unfortunately, ISO images aren't currently supported by the upload tool, although support is planned for the end of 2018.

In this tutorial, we'll demonstrate how to use the free and open-source VirtualBox virtualization tool to create a DigitalOcean-compatible VDI image (VirtualBox Disk Image) from an Ubuntu 18.04 ISO. The steps in this guide can be adapted to work with your preferred distribution's ISO images.

Prerequisites

Before you begin, you'll need the following available to you:

A local machine or remote server (with GUI access) onto which you'll install and use VirtualBox. In this tutorial we'll use a Mac OS X local machine, but you can use any system supported by VirtualBox. To learn more about supported systems, consult the VirtualBox Manual. The GUI menu options should be similar across operating systems, but may not be identical.
An ISO-format Ubuntu 18.04 Server OS image. The ubuntu-18.04.1-live-server-amd64.iso image meets the two requirements listed in the Custom Images Image Requirements:
- Your image must support the ext3 or ext4 filesystems
- Your image must have cloud-init 0.7.7, cloudbase-init, coreos-cloudinit, iginition, or bsd-cloudinit installed (Ubuntu 18.04 Server comes with cloud-init installed)

If you're adapting these steps for another distribution's ISO and your image does not have cloud-init installed and configured, you must install and configure it manually after installing the OS.

Once you have these prerequisites available to you, you're ready to begin with this guide.

Step 1 — Installing VirtualBox and Creating a Virtual Machine

The tool we'll use to convert the ISO-format image in this guide is VirtualBox, a free and open-source virtualizer for x86 hardware. By default, VirtualBox uses a GUI, which we'll use to create the VDI image in this guide.

To begin, download and install VirtualBox from the downloads page. Follow the appropriate link in the VirtualBox 5.2.20 platform packages section depending on your host operating system. In this guide, we'll be using an OSX system, so we'll download and install VirtualBox using the provided DMG.

Once you've installed VirtualBox, open the application.

You should see the following welcome screen:

Click on New to begin creating your Ubuntu virtual machine.

The following window should pop up, allowing you to name your virtual machine (VM) and select its OS:

In this tutorial, we'll name our VM Ubuntu 18.04, but feel free to give the VM a more descriptive name.

For Type, select Linux, and for Version, select Ubuntu (64-bit). Then, hit Continue.

The following screen should appear, allowing you to specify how much memory to allocate to your virtual machine:

Unless you have a more complex use case, 1024 MB should be enough memory for your virtual machine. If you need to adjust memory size, enter the amount of memory to be allocated to the VM, then hit Continue.

You should see the following screen:

This window allows you to create a virtual hard disk for your VM. This virtual hard disk is the image that you’ll upload to DigitalOcean in a later step. The Ubuntu operating system will be installed from the ISO you downloaded to this virtual hard disk. Make sure Create a virtual hard disk now is selected, and hit Create.

The following Hard disk file type window should appear, allowing you to select the format you'd like to use for your image:

All three types are supported by DigitalOcean Custom Images, so unless you have a strong preference, select VDI (VirtualBox Disk Image). Hit Continue.

You should then see the following window:

This window allows you to choose between a Dynamically allocated or Fixed size hard disk file. We'll use the default Dynamically allocated option and allow the file to grow as we install the Ubuntu OS and packages. Hit Continue.

The next window allows you to name your hard disk file (as well as choose the path to which it will be saved), and specify its maximum size:

Be sure to give yourself enough disk space to install the operating system as well as additional packages you may need. The default 10 GB should be fine for most purposes, but if you anticipate installing a large number of packages or storing a lot of data in the image, you should bump this up to your anticipated disk usage.

Once you've selected the size of the virtual hard disk, hit Create.

At this point, you'll be returned to the initial welcome screen, where you'll see the virtual machine you just created:

We can now begin installing Ubuntu onto the virtual machine.

Step 2 — Installing Ubuntu 18.04 onto the Virtual Machine

In this step we'll install and configure the Ubuntu operating system onto our virtual machine.

To begin, from the VirtualBox welcome screen, select your virtual machine, and hit the Start button in the toolbar.

You should see the following virtual machine window, prompting you to select the ISO file from which you'll boot the system:

Select the Ubuntu 18.04 Server ISO you downloaded, and hit Start.

In the VM, the Ubuntu installer will begin booting from the ISO, and you should be brought to the following menu:

Choose your preferred language using the arrow keys, and hit ENTER to continue.

You should then see the following Keyboard configuration screen:

Choose your preferred keyboard configuration, select Done, and hit ENTER.

Next, you'll be brought to the following installer selection screen:

Select Install Ubuntu, and hit ENTER.

The following Network connections screen should appear:

This screen allows you to configure the network interfaces for your Ubuntu server. Since we're performing the installation on a virtual machine, we'll just use the default option as the configured interface will be overwritten when we launch the image on the DigitalOcean platform.

Select Done and hit ENTER.

You'll then be brought to the following Configure proxy screen:

If you require a proxy, enter it here. Then, select Done, and hit ENTER.

The next screen will allow you to choose an Ubuntu archive mirror:

Unless you require a specific mirror, the default should be fine here. Select Done and hit ENTER.

Next, you'll be prompted to partition your virtual disk:

Unless you'd like to set up Logical Volume Manager (LVM) or manually partition the virtual disk, select Use An Entire Disk to use the entire attached virtual disk, and hit ENTER.

The following screen allows you to select the virtual disk that will be partitioned:

As described in the prompt text, the installer will create a partition for the bootloader, and use the remaining virtual disk space to create an ext4 partition to which the Ubuntu OS will be installed.

Select the attached virtual disk and hit ENTER.

The following screen displays a summary of the filesystem installer options before partitioning:

The ext4 partition will be mounted to /, and a second partition (1 MB) will be created for the GRUB bootloader. Once you've gone over and confirmed the partitioning scheme for your virtual disk, select Done and hit ENTER.

In the confirmation screen that appears, select Continue and hit ENTER.

The next screen will allow you to configure the system hostname, as well as an Ubuntu user:

Note that as you fill out this screen, the installer will continue copying files to the virtual disk in the background.

In this tutorial, we'll create a user named sammy and call our server ubuntu. The server name will likely be overwritten when this image is run on the DigitalOcean platform, so feel free to give it a temporary name here.

You can upload your SSH keys to DigitalOcean and automatically embed them into created Droplets, so for now we won't Import SSH identity. To learn how to upload your SSH keys to DigitalOcean, consult the Droplet Product Documentation.

Once you've filled in all the required fields, the prompt should look something like this:

Select Done and hit ENTER.

The next screen will prompt you to select popular snaps for your Ubuntu server. Snaps are prepackaged bundles of software that contain an application, its dependencies, and configuration. To learn more about snaps, consult the Snap Documentation.

In this guide we won't install any snaps and will manually install packages in a later step. If you'd like to install a snap, select or deselect it using SPACE and scroll down to Done. Then, hit ENTER.

Regardless of your selection in the snap screen, you'll then be brought to an installation progress and summary screen:

Once the installation completes, select Reboot Now and hit ENTER.

The installer will shut down and prompt you to remove the installation medium (in this case this is the ISO image we selected earlier). In most cases, the ISO will be detached automatically upon reboot, so you can simply hit ENTER.

To double check, in the VirtualBox GUI menu, navigate to Devices, and then Optical Drives. If the Remove disk from virtual drive option is available to you, click on it to detach the ISO from the virtual machine. Then, back in the virtual machine window, hit ENTER.

The system will reboot in the virtual machine, this time from the virtual disk to which we installed Ubuntu.

Since cloud-init is installed by default on Ubuntu 18.04 Server, the first time Ubuntu boots, cloud-init will run and configure itself. In the virtual machine window, you should see some cloud-init log items and have a prompt available to you. Hit ENTER.

You can then log in to your Ubuntu server using the user you created in the installer.

Enter your username and hit ENTER, then enter your password and hit ENTER.

You should now have access to a command prompt, indicating that you've successfully completed the Ubuntu 18.04 installation, and are now logged in as the user you created previously.

In the next step of this guide, we'll reconfigure cloud-init and set it up to run when the Ubuntu image is launched as a Droplet on the DigitalOcean platform.

Step 3 — Reconfiguring `cloud-init`

Now that we've installed Ubuntu 18.04 to a virtual disk and have the system up and running, we need to reconfigure cloud-init to use the appropriate datasource for the DigitalOcean platform. A cloud-init datasource is a source of config data for cloud-init that typically consists of userdata (like shell scripts) or server metadata, like hostname, instance-id, etc. To learn more about cloud-init datasources, consult the official cloud-init docs.

By default, on Ubuntu 18.04, cloud-init configures itself to use the DataSourceNoCloud datasource. This will cause problems when running the image on DigitalOcean, so we need to reconfigure cloud-init to use the ConfigDrive datasource and ensure that cloud-init reruns when the image is launched on DigitalOcean.

To begin, ensure that you've started your Ubuntu 18.04 virtual machine and have logged in as the user you created earlier.

From the command line, navigate to the /etc/cloud/cloud.cfg.d directory:

cd /etc/cloud/cloud.cfg.d

Use the ls command to list the cloud-init config files present in the directory:

ls

Output05_logging.cfg  50-curtin-networking.cfg  90_dpkg.cfg  curtin-preserve-sources.cfg  README

First, delete the 50-curtin-networking.cfg file, which configures networking interfaces for your Ubuntu server. When the image is launched on DigitalOcean, cloud-init will run and reconfigure these interfaces automatically. If this file is not deleted, the DigitalOcean Droplet created from this Ubuntu image will have its interfaces misconfigured and won't be accessible from the internet.

sudo rm 50-curtin-networking.cfg

Next, we'll run dpkg-reconfigure cloud-init to remove the NoCloud datasource, ensuring that cloud-init searches for and finds the ConfigDrive datasource used on DigitalOcean:

sudo dpkg-reconfigure cloud-init

You should see the following graphical menu:

The NoCloud datasource is initially highlighted. Press SPACE to unselect it, then hit ENTER.

Finally, navigate to /etc/netplan:

cd /etc/netplan

Remove the 50-cloud-init.yaml file (this was generated from the cloud-init networking file we removed earlier):

sudo rm 50-cloud-init.yaml

The final step is ensuring that we clean up configuration from the initial cloud-init run so that it reruns when the image is launched on DigitalOcean.

To do this, run cloud-init clean:

sudo cloud-init clean

At this point, your image is ready to be launched on the DigitalOcean platform. You can install additional packages and software into your image. Once you're done, shutdown your virtual machine:

sudo shutdown -h now

We can now move on to uploading and launching this custom image on the DigitalOcean platform.

Step 4 — Uploading Custom Image and Creating Droplet

Now that we've created an Ubuntu 18.04 VDI image and configured it for use on DigitalOcean, we can upload it using the Custom Images upload tool.

On macOS, the Ubuntu virtual disk image we created and configured will be located by default at ~/VirtualBox VMs/your_VM_name/your_virtual_disk_name.vdi. This path may vary slightly depending on the OS you're using with VirtualBox.

Before we upload the image, we'll compress it to speed up the file transfer to DigitalOcean.

On your host OS (not inside the virtual machine), navigate to the directory containing your VDI image file:

cd ~/VirtualBox\ VMs/Ubuntu\ 18.04/

Now, use gzip to compress the file:

gzip < Ubuntu\ 18.04.vdi > Ubuntu\ 18.04.gz

In this command we pipe the source Ubuntu 18.04.vdi file into gzip, specifying as output the Ubuntu 18.04.gz compressed file.

Once gzip finishes compressing your file, upload the .gz file to DigitalOcean, following instructions in the Custom Images Quickstart.

You should now be able to create and use Droplets from your custom Ubuntu 18.04 Server image.

Conclusion

In this tutorial, we learned how to create a custom VDI image from a vanilla Ubuntu 18.04 ISO using the VirtualBox virtualization tool. We adjusted cloud-init so it can properly configure Droplet networking on DigitalOcean, and finally compressed and uploaded the image using the Custom Images upload tool.

You can adjust the steps in this tutorial to work with your preferred Linux distribution’s ISO images. Ensure that you have an SSH server installed and configured to start on boot, and that cloud-init has been installed and properly configured to use the ConfigDrive datasource. Finally, ensure that any stale networking configuration files have been purged.

You may also wish to use a tool like Packer to automate the creation of your machine images.

To learn more about DigitalOcean Custom Images, consult the Custom Images product docs and launch blog post.

How to Deploy a Symfony 4 Application to Production with LEMP on Ubuntu 18.04

2018-10-16T16:46:49Z

The author selected Software in the Public Interest Inc to receive a donation as part of the Write for DOnations program.

Introduction

Symfony is an open-source PHP framework with an elegant structure and a reputation for being a suitable framework to kick-start any project irrespective of its size. As a set of reusable components, its flexibility, architecture, and high performance make it a top choice for building a highly complex enterprise application.

In this tutorial, you will deploy an existing, standard Symfony 4 application to production with a LEMP stack (Nginx, MySQL, and PHP) on Ubuntu 18.04, which will help you get started configuring the server and the structure of the framework. Nginx is a popular open-source, high-performance HTTP server with additional features including reverse proxy support. It has a good reputation and hosts some of the largest and highest traffic sites on the internet. If you choose to deploy your own Symfony application instead, you might have to implement extra steps depending on the existing structure of your application.

Prerequisites

To complete this tutorial, you will need:

One Ubuntu 18.04 server set up by following the Ubuntu 18.04 initial server setup guide, including a non-root user with sudo access and a firewall.
Nginx, MySQL, and PHP installed by following How To Install Linux, Nginx, MySQL, PHP (LEMP stack) on Ubuntu 18.04 .
Composer installed following steps 1 and 2 of How to Install and Use Composer on Ubuntu 18.04.
Git installed by following How to Install Git on Ubuntu 18.04.

Step 1 — Creating a User and Database for the Application

By following the instructions in the Prerequisites, you now have all the basic server dependencies required for the application installation. As every dynamic web application requires a database, you will create a user and properly configure a database for the application in this section.

To create a MySQL database for our application and a user associated with it, you need to access the MySQL client using the MySQL root account:

 mysql -u root -p

Enter the appropriate password, which should be the same password used when running mysql_secure_installation.

Next, create the application database with:

CREATE DATABASE blog;

You will see the following output in the console:

OutputQuery OK, 1 row affected (0.00 sec)

You have successfully created your application database. You can now create a MySQL user and grant them access to the newly created database.

Execute the following command to create a MySQL user and password. You can change the username and password to something more secure if you wish:

CREATE USER  'blog-admin'@'localhost' IDENTIFIED BY 'password';

You will see the following output:

OutputQuery OK, 0 rows affected (0.00 sec)

Currently, the user blog-admin does not have the right permission over the application database. In fact, even if blog-admin tries to log-in with their password, they will not be able to reach the MySQL shell.

A user needs the right permission before accessing or carrying out a specific action on a database. Use the following command to allow complete access to the blog database for the blog-admin user:

GRANT ALL PRIVILEGES ON blog.* TO 'blog-admin'@'localhost';

You will see the following output:

OutputQuery OK, 0 rows affected (0.00 sec)

The blog-admin now has all privileges on all the tables inside the blog database. To reload the grant tables and apply changes, you need to perform a flush-privilege operation using the flush statement:

FLUSH PRIVILEGES;

You will see the following output:

OutputQuery OK, 0 rows affected (0.00 sec)

You are done creating a new user and granting privileges. To test if you’re on track, exit the MySQL client:

quit;

And log in again, using the credentials of the MySQL user you just created and enter the password when prompted:

mysql -u blog-admin -p

Check that the database can be accessed by the user with:

SHOW DATABASES;

You'll see the blog table in the output:

Output+--------------------+
| Database           |
+--------------------+
| information_schema |
| blog               |
+--------------------+
2 rows in set (0.00 sec)

Finally, exit the MySQL client:

quit;

You have successfully created a database, a user for the demo application, and granted the newly created user the right privileges to access the database. You are now ready to set up the demo application.

Step 2 — Setting Up the Demo Application

To keep this tutorial simple, you will deploy a blog application built with Symfony. This application will allow an authenticated user to create a blog post and store it in the database. In addition, the application user can view all the posts and details associated with an author.

The source code of the blog application you will deploy in this tutorial is on GitHub. You will use Git to pull the source code of the application from GitHub and save it in a new directory.

First, create a directory that will serve as the root directory for your application. So, run the following command from the console to create a new directory named symfony-blog:

sudo mkdir -p /var/www/symfony-blog

In order to work with the project files using a non-root user account, you’ll need to change the folder owner and group by running:

 sudo chown sammy:sammy /var/www/symfony-blog

Replace sammy with your sudo non-root username.

Now, you can change into the parent directory and clone the application on GitHub:

cd /var/www
git clone https://github.com/yemiwebby/symfony-blog.git symfony-blog

You'll see the following output:

OutputCloning into 'symfony-blog'...
remote: Counting objects: 180, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 180 (delta 57), reused 164 (delta 41), pack-reused 0
Receiving objects: 100% (180/180), 167.01 KiB | 11.13 MiB/s, done.
Resolving deltas: 100% (57/57), done.

The demo application is now set. In the next step, you will configure the environment variables and install the required dependencies for the project.

Step 3 — Configuring your Environment Variables for the Application

To completely set up the application, you need to install the project dependencies and properly configure the application parameters.

By default, the Symfony application runs in a development mode, which gives it a very detailed log for the purposes of debugging. This is not applicable to what you are doing in this tutorial, and not good practice for a production environment, as it can slow things down and create very large log files.

Symfony needs to be aware that you’re running the application in a production environment. You can set this up by either creating a .env file containing variable declarations, or creating environment variables directly. Since you can also use the .env file to configure your database credentials for this application, it makes more sense for you to do this. Change your working directory to the cloned project and create the .env file with:

cd symfony-blog
sudo nano .env

Add the following lines to the file to configure the production application environment:

.env

APP_ENV=prod
APP_DEBUG=0

APP_ENV is an environment variable that specifies that the application is in production, while APP_DEBUG is an environment variable that specifies if the application should run in debug mode or not. You have set it to false for now.

Save the file and exit the editor.

Next, install a PHP extension that Symfony apps use to handle XML:

sudo apt install php7.2-xml

Next, you need to install the project dependencies, run composer install:

cd /var/www/symfony-blog
composer install

You have successfully configured the environment variables and installed the required dependencies for the project. Next, you will set up the database credentials.

Step 4 — Setting Up Database Credentials

In order to retrieve data from the application’s database you created earlier, you will need to set up and configure the required database credentials from within the Symfony application.

Open the .env file again:

sudo nano .env

Add the following content to the file, which will allow you to easily connect and interact properly with the database. You can add it right after the APP_DEBUG=0 line within the .env file:

.env

...
DATABASE_URL=mysql://blog-admin:password@localhost:3306/blog

The Symfony framework uses a third-party library called Doctrine to communicate with databases. Doctrine gives you useful tools to make interactions with databases easy and flexible.

You can now use Doctrine to update your database with the tables from the cloned Github application. Run this command to do that:

php bin/console doctrine:schema:update --force

You'll see the following output:

OutputUpdating database schema...
    4 queries were executed
[OK] Database schema updated successfully!

After setting up the required credentials and updating the database schema, you can now easily interact with the database. In order to start the application with some data, you will load a set of dummy data into the database in the next section.

Step 5 — Populating your Database Using Doctrine-Fixtures

At the moment, the newly created tables are empty. You will populate it using doctrine-fixtures. Using Doctrine-Fixtures is not a prerequisite for Symfony applications, it is only used to provide dummy data for your application.

Run the following command to automatically load testing data that contains the details of an author and a sample post into the database table created for the blog:

php bin/console doctrine:fixtures:load

You will get a warning about the database getting purged. You can go ahead and type Y:

OutputCareful, database will be purged. Do you want to continue y/N ? y
  > purging database
  > loading App\DataFixtures\ORM\Fixtures

In the next section you will clear and warm up you cache.

Step 6 — Clearing and Warming Up your Cache

To ensure your application loads faster when users make requests, it is good practice to warm the cache during the deployment. Warming up the cache generates pages and stores them for faster responses later rather than building completely new pages. Fortunately, Symfony has a command to clear the cache that also triggers a warm up. Run the following command for that purpose:

php bin/console cache:clear

You will see the following output:

OutputClearing the cache for the prod environment with debug false
[OK] Cache for the "prod" environment (debug=false) was successfully cleared.

You will conclude the set up in a bit. All that remains is to configure the web server. You will do that in the next section.

Step 7 — Configuring the Web Server and Running the Application

By now, you have Nginx installed to serve your pages and MySQL to store and manage your data. You will now configure the web server by creating a new application server block, instead of editing the default one.

Open a new server block with:

sudo nano /etc/nginx/sites-available/blog

Add the following content to the new server block configuration file. Ensure you replace the your_server_ip within the server block with your server IP address:

/etc/nginx/sites-available/blog


server {
    listen 80;
    listen [::]:80;

    server_name blog your_server_ip;
    root /var/www/symfony-blog/public;
    index index.php;
    client_max_body_size 100m;

    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location ~ \.php {
        try_files $uri /index.php =404;
        fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param SCRIPT_NAME $fastcgi_script_name;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_index index.php;
        include fastcgi_params;
      }

    location ~ /\.(?:ht|git|svn) {
        deny all;
    }
}

First, we specified the listen directives for Nginx, which is by default on port 80, and then set the server name to match requests for the server’s IP address. Next, we used the root directives to specify the document root for the project. The symfony-blog application is stored in /var/www/symfony-blog, but to comply with best practices, we set the web root to /var/www/symfony-blog/public as only the /public subdirectory should be exposed to the internet. Finally, we configured the location directive to handle PHP processing.

After adding the content, save the file and exit the editor.

Note: If you created the file example.com in the prerequisite article How To Install Linux, Nginx, MySQL, PHP (LEMP stack) on Ubuntu 18.04, remove it from the sites-enabled directory with sudo rm /etc/nginx/sites-enabled/example.com so it doesn't conflict with this new file.

To enable the newly created server block, we need to create a symbolic link from the new server block configuration file located in /etc/nginx/sites-available directory to the /etc/nginx/sites-enabled by using the following command:

sudo ln -s /etc/nginx/sites-available/blog /etc/nginx/sites-enabled/

Check the new configuration file for any syntax errors by running:

sudo nginx -t

This command will print errors to the console if there are any. Once there are no errors run this command to reload Nginx:

 sudo systemctl reload nginx

You just concluded the last step required to successfully deploy the Symfony 4 application. You configured the web server by creating a server block and properly set the web root in order to make the web application accessible.

Finally, you can now run and test out the application. Visit http://your_server_ip in your favorite browser:

The following image is the screenshot of the Symfony blog application that you should see at your server's IP address:

Conclusion

Symfony is a feature-rich PHP framework with an architecture that makes web development fun for the developer who builds software using it. Symfony is a feature-rich web development framework that provides developers powerful tools to build web applications. It's often considered a good choice for enterprise applications due to its flexibility. The steps to deploy a typical Symfony application vary—depending on the setup, complexity, and the requirements of the application.

In this tutorial, you manually deployed a Symfony 4 application to production on an Ubuntu 18.04 server running LEMP. You can now apply this knowledge to deploying your own Symfony applications.

An Introduction to Queries in MySQL

2018-10-17T19:05:28Z

Introduction

Databases are a key component of many websites and applications, and are at the core of how data is stored and exchanged across the internet. One of the most important aspects of database management is the practice of retrieving data from a database, whether it's on an ad hoc basis or part of a process that's been coded into an application. There are several ways to retrieve information from a database, but one of the most commonly-used methods is performed through submitting queries through the command line.

In relational database management systems, a query is any command used to retrieve data from a table. In Structured Query Language (SQL), queries are almost always made using the SELECT statement.

In this guide, we will discuss the basic syntax of SQL queries as well as some of the more commonly-employed functions and operators. We will also practice making SQL queries using some sample data in a MySQL database.

MySQL is an open-source relational database management system. One of the most widely-deployed SQL-databases, MySQL prioritizes speed, reliability, and usability. It generally follows the ANSI SQL standard, although there are a few cases where MySQL performs operations differently than the recognized standard.

Prerequisites

In general, the commands and concepts presented in this guide can be used on any Linux-based operating system running any SQL database software. However, it was written specifically with an Ubuntu 18.04 server running MySQL in mind. To set this up, you will need the following:

An Ubuntu 18.04 machine with a non-root user with sudo privileges. This can be set up using our Initial Server Setup guide for Ubuntu 18.04.
MySQL installed on the machine. Our guide on How to Install MySQL on Ubuntu 18.04 can help you set this up.

With this setup in place, we can begin the tutorial.

Creating a Sample Database

Before we can begin making queries in SQL, we will first create a database and a couple tables, then populate these tables with some sample data. This will allow you to gain some hands-on experience when you begin making queries later on.

For the sample database we'll use throughout this guide, imagine the following scenario:

You and several of your friends all celebrate your birthdays with one another. On each occasion, the members of the group head to the local bowling alley, participate in a friendly tournament, and then everyone heads to your place where you prepare the birthday-person's favorite meal.

Now that this tradition has been going on for a while, you've decided to begin tracking the records from these tournaments. Also, to make planning dinners easier, you decide to create a record of your friends' birthdays and their favorite entrees, sides, and desserts. Rather than keep this information in a physical ledger, you decide to exercise your database skills by recording it in a MySQL database.

To begin, open up a MySQL prompt as your root MySQL user:

sudo mysql

Note: If you followed the prerequisite the tutorial on Installing MySQL on Ubuntu 18.04, you may have configured your root user to authenticate using a password. In this case, you will connect to the MySQL prompt with the following command:

mysql -u root -p

Next, create the database by running:

CREATE DATABASE `birthdays`;

Then select this database by typing:

USE birthdays;

Next, create two tables within this database. We'll use the first table to track your friends' records at the bowling alley. The following command will create a table called tourneys with columns for the name of each of your friends, the number of tournaments they've won (wins), their all-time best score, and what size bowling shoe they wear (size):

CREATE TABLE tourneys ( 
name varchar(30), 
wins real, 
best real, 
size real 
);

Once you run the CREATE TABLE command and populate it with column headings, you’ll receive the following output:

OutputQuery OK, 0 rows affected (0.00 sec)

Populate the tourneys table with some sample data:

INSERT INTO tourneys (name, wins, best, size) 
VALUES ('Dolly', '7', '245', '8.5'), 
('Etta', '4', '283', '9'), 
('Irma', '9', '266', '7'), 
('Barbara', '2', '197', '7.5'), 
('Gladys', '13', '273', '8');

You’ll receive an output like this:

OutputQuery OK, 5 rows affected (0.01 sec)
Records: 5  Duplicates: 0  Warnings: 0

Following this, create another table within the same database which we'll use to store information about your friends' favorite birthday meals. The following command creates a table named dinners with columns for the name of each of your friends, their birthdate, their favorite entree, their preferred side dish, and their favorite dessert:

CREATE TABLE dinners ( 
name varchar(30), 
birthdate date, 
entree varchar(30), 
side varchar(30), 
dessert varchar(30) 
);

Similarly for this table, you’ll receive feedback confirming that the command ran successfully:

OutputQuery OK, 0 rows affected (0.01 sec)

Populate this table with some sample data as well:

INSERT INTO dinners (name, birthdate, entree, side, dessert) 
VALUES ('Dolly', '1946-01-19', 'steak', 'salad', 'cake'), 
('Etta', '1938-01-25', 'chicken', 'fries', 'ice cream'), 
('Irma', '1941-02-18', 'tofu', 'fries', 'cake'), 
('Barbara', '1948-12-25', 'tofu', 'salad', 'ice cream'), 
('Gladys', '1944-05-28', 'steak', 'fries', 'ice cream');

OutputQuery OK, 5 rows affected (0.00 sec)
Records: 5  Duplicates: 0  Warnings: 0

Once that command completes successfully, you're done setting up your database. Next, we'll go over the basic command structure of SELECT queries.

Understanding SELECT Statements

As mentioned in the introduction, SQL queries almost always begin with the SELECT statement. SELECT is used in queries to specify which columns from a table should be returned in the result-set. Queries also almost always include FROM, which is used to specify which table the statement will query.

Generally, SQL queries follow this syntax:

SELECT column_to_select FROM table_to_select WHERE certain_conditions_apply;

By way of example, the following statement will return the entire name column from the dinners table:

SELECT name FROM dinners;

Output+---------+
| name    |
+---------+
| Dolly   |
| Etta    |
| Irma    |
| Barbara |
| Gladys  |
+---------+
5 rows in set (0.00 sec)

You can select multiple columns from the same table by separating their names with a comma, like this:

SELECT name, birthdate FROM dinners;

Output+---------+------------+
| name    | birthdate  |
+---------+------------+
| Dolly   | 1946-01-19 |
| Etta    | 1938-01-25 |
| Irma    | 1941-02-18 |
| Barbara | 1948-12-25 |
| Gladys  | 1944-05-28 |
+---------+------------+
5 rows in set (0.00 sec)

Instead of naming a specific column or set of columns, you can follow the SELECT operator with an asterisk (*) which serves as a placeholder representing all the columns in a table. The following command returns every column from the tourneys table:

SELECT * FROM tourneys;

Output+---------+------+------+------+
| name    | wins | best | size |
+---------+------+------+------+
| Dolly   |    7 |  245 |  8.5 |
| Etta    |    4 |  283 |    9 |
| Irma    |    9 |  266 |    7 |
| Barbara |    2 |  197 |  7.5 |
| Gladys  |   13 |  273 |    8 |
+---------+------+------+------+
5 rows in set (0.00 sec)

WHERE is used in queries to filter records that meet a specified condition, and any rows that do not meet that condition are eliminated from the result. A WHERE clause typically follows this syntax:

. . . WHERE column_name comparison_operator value

The comparison operator in a WHERE clause defines how the specified column should be compared against the value. Here are some common SQL comparison operators:

Operator	What it does
`=`	tests for equality
`!=`	tests for inequality
`<`	tests for less-than
`>`	tests for greater-than
`<=`	tests for less-than or equal-to
`>=`	tests for greater-than or equal-to
`BETWEEN`	tests whether a value lies within a given range
`IN`	tests whether a row's value is contained in a set of specified values
`EXISTS`	tests whether rows exist, given the specified conditions
`LIKE`	tests whether a value matches a specified string
`IS NULL`	tests for `NULL` values
`IS NOT NULL`	tests for all values other than `NULL`

For example, if you wanted to find Irma's shoe size, you could use the following query:

SELECT size FROM tourneys WHERE name = 'Irma';

Output+------+
| size |
+------+
|    7 |
+------+
1 row in set (0.00 sec)

SQL allows the use of wildcard characters, and these are especially handy when used in WHERE clauses. Percentage signs (%) represent zero or more unknown characters, and underscores (_) represent a single unknown character. These are useful if you're trying to find a specific entry in a table, but aren't sure of what that entry is exactly. To illustrate, let's say that you've forgotten the favorite entree of a few of your friends, but you're certain this particular entree starts with a "t." You could find its name by running the following query:

SELECT entree FROM dinners WHERE entree LIKE 't%';

Output+--------+
| entree |
+--------+
| tofu   |
| tofu   |
+--------+
2 rows in set (0.00 sec)

Based on the output above, we see that the entree we have forgotten is tofu.

There may be times when you're working with databases that have columns or tables with relatively long or difficult-to-read names. In these cases, you can make these names more readable by creating an alias with the AS keyword. Aliases created with AS are temporary, and only exist for the duration of the query for which they're created:

SELECT name AS n, birthdate AS b, dessert AS d FROM dinners;

Output+---------+------------+-----------+
| n       | b          | d         |
+---------+------------+-----------+
| Dolly   | 1946-01-19 | cake      |
| Etta    | 1938-01-25 | ice cream |
| Irma    | 1941-02-18 | cake      |
| Barbara | 1948-12-25 | ice cream |
| Gladys  | 1944-05-28 | ice cream |
+---------+------------+-----------+
5 rows in set (0.00 sec)

Here, we have told SQL to display the name column as n, the birthdate column as b, and the dessert column as d.

The examples we've gone through up to this point include some of the more frequently-used keywords and clauses in SQL queries. These are useful for basic queries, but they aren't helpful if you're trying to perform a calculation or derive a scalar value (a single value, as opposed to a set of multiple different values) based on your data. This is where aggregate functions come into play.

Aggregate Functions

Oftentimes, when working with data, you don't necessarily want to see the data itself. Rather, you want information about the data. The SQL syntax includes a number of functions that allow you to interpret or run calculations on your data just by issuing a SELECT query. These are known as aggregate functions.

The COUNT function counts and returns the number of rows that match a certain criteria. For example, if you'd like to know how many of your friends prefer tofu for their birthday entree, you could issue this query:

SELECT COUNT(entree) FROM dinners WHERE entree = 'tofu';

Output+---------------+
| COUNT(entree) |
+---------------+
|             2 |
+---------------+
1 row in set (0.00 sec)

The AVG function returns the average (mean) value of a column. Using our example table, you could find the average best score amongst your friends with this query:

SELECT AVG(best) FROM tourneys;

Output+-----------+
| AVG(best) |
+-----------+
|     252.8 |
+-----------+
1 row in set (0.00 sec)

SUM is used to find the total sum of a given column. For instance, if you'd like to see how many games you and your friends have bowled over the years, you could run this query:

SELECT SUM(wins) FROM tourneys;

Output+-----------+
| SUM(wins) |
+-----------+
|        35 |
+-----------+
1 row in set (0.00 sec)

Note that the AVG and SUM functions will only work correctly when used with numeric data. If you try to use them on non-numerical data, it will result in either an error or just 0, depending on which RDBMS you're using:

SELECT SUM(entree) FROM dinners;

Output+-------------+
| SUM(entree) |
+-------------+
|           0 |
+-------------+
1 row in set, 5 warnings (0.00 sec)

MIN is used to find the smallest value within a specified column. You could use this query to see what the worst overall bowling record is so far (in terms of number of wins):

SELECT MIN(wins) FROM tourneys;

Output+-----------+
| MIN(wins) |
+-----------+
|         2 |
+-----------+
1 row in set (0.00 sec)

Similarly, MAX is used to find the largest numeric value in a given column. The following query will show the best overall bowling record:

SELECT MAX(wins) FROM tourneys;

Output+-----------+
| MAX(wins) |
+-----------+
|        13 |
+-----------+
1 row in set (0.00 sec)

Unlike SUM and AVG, the MIN and MAX functions can be used for both numeric and alphabetic data types. When run on a column containing string values, the MIN function will show the first value alphabetically:

SELECT MIN(name) FROM dinners;

Output+-----------+
| MIN(name) |
+-----------+
| Barbara   |
+-----------+
1 row in set (0.00 sec)

Likewise, when run on a column containing string values, the MAX function will show the last value alphabetically:

SELECT MAX(name) FROM dinners;

Output+-----------+
| MAX(name) |
+-----------+
| Irma      |
+-----------+
1 row in set (0.00 sec)

Aggregate functions have many uses beyond what was described in this section. They're particularly useful when used with the GROUP BY clause, which is covered in the next section along with several other query clauses that affect how result-sets are sorted.

Manipulating Query Outputs

In addition to the FROM and WHERE clauses, there are several other clauses which are used to manipulate the results of a SELECT query. In this section, we will explain and provide examples for some of the more commonly-used query clauses.

One of the most frequently-used query clauses, aside from FROM and WHERE, is the GROUP BY clause. It's typically used when you're performing an aggregate function on one column, but in relation to matching values in another.

For example, let's say you wanted to know how many of your friends prefer each of the three entrees you make. You could find this info with the following query:

SELECT COUNT(name), entree FROM dinners GROUP BY entree;

Output+-------------+---------+
| COUNT(name) | entree  |
+-------------+---------+
|           1 | chicken |
|           2 | steak   |
|           2 | tofu    |
+-------------+---------+
3 rows in set (0.00 sec)

The ORDER BY clause is used to sort query results. By default, numeric values are sorted in ascending order, and text values are sorted in alphabetical order. To illustrate, the following query lists the name and birthdate columns, but sorts the results by birthdate:

SELECT name, birthdate FROM dinners ORDER BY birthdate;

Output+---------+------------+
| name    | birthdate  |
+---------+------------+
| Etta    | 1938-01-25 |
| Irma    | 1941-02-18 |
| Gladys  | 1944-05-28 |
| Dolly   | 1946-01-19 |
| Barbara | 1948-12-25 |
+---------+------------+
5 rows in set (0.00 sec)

Notice that the default behavior of ORDER BY is to sort the result-set in ascending order. To reverse this and have the result-set sorted in descending order, close the query with DESC:

SELECT name, birthdate FROM dinners ORDER BY birthdate DESC;

Output+---------+------------+
| name    | birthdate  |
+---------+------------+
| Barbara | 1948-12-25 |
| Dolly   | 1946-01-19 |
| Gladys  | 1944-05-28 |
| Irma    | 1941-02-18 |
| Etta    | 1938-01-25 |
+---------+------------+
5 rows in set (0.00 sec)

As mentioned previously, the WHERE clause is used to filter results based on specific conditions. However, if you use the WHERE clause with an aggregate function, it will return an error, as is the case with the following attempt to find which sides are the favorite of at least three of your friends:

SELECT COUNT(name), side FROM dinners WHERE COUNT(name) >= 3;

OutputERROR 1111 (HY000): Invalid use of group function

The HAVING clause was added to SQL to provide functionality similar to that of the WHERE clause while also being compatible with aggregate functions. It's helpful to think of the difference between these two clauses as being that WHERE applies to individual records, while HAVING applies to group records. To this end, any time you issue a HAVING clause, the GROUP BY clause must also be present.

The following example is another attempt to find which side dishes are the favorite of at least three of your friends, although this one will return a result without error:

SELECT COUNT(name), side FROM dinners GROUP BY side HAVING COUNT(name) >= 3;

Output+-------------+-------+
| COUNT(name) | side  |
+-------------+-------+
|           3 | fries |
+-------------+-------+
1 row in set (0.00 sec)

Aggregate functions are useful for summarizing the results of a particular column in a given table. However, there are many cases where it's necessary to query the contents of more than one table. We'll go over a few ways you can do this in the next section.

Querying Multiple Tables

More often than not, a database contains multiple tables, each holding different sets of data. SQL provides a few different ways to run a single query on multiple tables.

The JOIN clause can be used to combine rows from two or more tables in a query result. It does this by finding a related column between the tables and sorts the results appropriately in the output.

SELECT statements that include a JOIN clause generally follow this syntax:

SELECT table1.column1, table2.column2
FROM table1
JOIN table2 ON table1.related_column=table2.related_column;

Note that because JOIN clauses compare the contents of more than one table, the previous example specifies which table to select each column from by preceding the name of the column with the name of the table and a period. You can specify which table a column should be selected from like this for any query, although it's not necessary when selecting from a single table, as we've done in the previous sections. Let's walk through an example using our sample data.

Imagine that you wanted to buy each of your friends a pair of bowling shoes as a birthday gift. Because the information about your friends' birthdates and shoe sizes are held in separate tables, you could query both tables separately then compare the results from each. With a JOIN clause, though, you can find all the information you want with a single query:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
JOIN dinners ON tourneys.name=dinners.name;

Output+---------+------+------------+
| name    | size | birthdate  |
+---------+------+------------+
| Dolly   |  8.5 | 1946-01-19 |
| Etta    |    9 | 1938-01-25 |
| Irma    |    7 | 1941-02-18 |
| Barbara |  7.5 | 1948-12-25 |
| Gladys  |    8 | 1944-05-28 |
+---------+------+------------+
5 rows in set (0.00 sec)

The JOIN clause used in this example, without any other arguments, is an inner JOIN clause. This means that it selects all the records that have matching values in both tables and prints them to the results set, while any records that aren't matched are excluded. To illustrate this idea, let's add a new row to each table that doesn't have a corresponding entry in the other:

INSERT INTO tourneys (name, wins, best, size) 
VALUES ('Bettye', '0', '193', '9');

INSERT INTO dinners (name, birthdate, entree, side, dessert) 
VALUES ('Lesley', '1946-05-02', 'steak', 'salad', 'ice cream');

Then, re-run the previous SELECT statement with the JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
JOIN dinners ON tourneys.name=dinners.name;

Output+---------+------+------------+
| name    | size | birthdate  |
+---------+------+------------+
| Dolly   |  8.5 | 1946-01-19 |
| Etta    |    9 | 1938-01-25 |
| Irma    |    7 | 1941-02-18 |
| Barbara |  7.5 | 1948-12-25 |
| Gladys  |    8 | 1944-05-28 |
+---------+------+------------+
5 rows in set (0.00 sec)

Notice that, because the tourneys table has no entry for Lesley and the dinners table has no entry for Bettye, those records are absent from this output.

It is possible, though, to return all the records from one of the tables using an outer JOIN clause. In MySQL, JOIN clauses are written as either LEFT JOIN or RIGHT JOIN.

A LEFT JOIN clause returns all the records from the “left” table and only the matching records from the right table. In the context of outer joins, the left table is the one referenced by the FROM clause, and the right table is any other table referenced after the JOIN statement.

Run the previous query again, but this time use a LEFT JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
LEFT JOIN dinners ON tourneys.name=dinners.name;

This command will return every record from the left table (in this case, tourneys) even if it doesn't have a corresponding record in the right table. Any time there isn't a matching record from the right table, it's returned as NULL or just a blank value, depending on your RDBMS:

Output+---------+------+------------+
| name    | size | birthdate  |
+---------+------+------------+
| Dolly   |  8.5 | 1946-01-19 |
| Etta    |    9 | 1938-01-25 |
| Irma    |    7 | 1941-02-18 |
| Barbara |  7.5 | 1948-12-25 |
| Gladys  |    8 | 1944-05-28 |
| Bettye  |    9 | NULL       |
+---------+------+------------+
6 rows in set (0.00 sec)

Now run the query again, this time with a RIGHT JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
RIGHT JOIN dinners ON tourneys.name=dinners.name;

This will return all the records from the right table (dinners). Because Lesley's birthdate is recorded in the right table, but there is no corresponding row for her in the left table, the name and size columns will return as NULL values in that row:

Output+---------+------+------------+
| name    | size | birthdate  |
+---------+------+------------+
| Dolly   |  8.5 | 1946-01-19 |
| Etta    |    9 | 1938-01-25 |
| Irma    |    7 | 1941-02-18 |
| Barbara |  7.5 | 1948-12-25 |
| Gladys  |    8 | 1944-05-28 |
| NULL    | NULL | 1946-05-02 |
+---------+------+------------+
6 rows in set (0.00 sec)

Note that left and right joins can be written as LEFT OUTER JOIN or RIGHT OUTER JOIN, although the OUTER part of the clause is implied. Likewise, specifying INNER JOIN will produce the same result as just writing JOIN.

As an alternative to using JOIN to query records from multiple tables, you can use the UNION clause.

The UNION operator works slightly differently than a JOIN clause: instead of printing results from multiple tables as unique columns using a single SELECT statement, UNION combines the results of two SELECT statements into a single column.

To illustrate, run the following query:

SELECT name FROM tourneys UNION SELECT name FROM dinners;

This query will remove any duplicate entries, which is the default behavior of the UNION operator:

Output+---------+
| name    |
+---------+
| Dolly   |
| Etta    |
| Irma    |
| Barbara |
| Gladys  |
| Bettye  |
| Lesley  |
+---------+
7 rows in set (0.00 sec)

To return all entries (including duplicates) use the UNION ALL operator:

SELECT name FROM tourneys UNION ALL SELECT name FROM dinners;

Output+---------+
| name    |
+---------+
| Dolly   |
| Etta    |
| Irma    |
| Barbara |
| Gladys  |
| Bettye  |
| Dolly   |
| Etta    |
| Irma    |
| Barbara |
| Gladys  |
| Lesley  |
+---------+
12 rows in set (0.00 sec)

The names and number of the columns in the results table reflect the name and number of columns queried by the first SELECT statement. Note that when using UNION to query multiple columns from more than one table, each SELECT statement must query the same number of columns, the respective columns must have similar data types, and the columns in each SELECT statement must be in the same order. The following example shows what might result if you use a UNION clause on two SELECT statements that query a different number of columns:

SELECT name FROM dinners UNION SELECT name, wins FROM tourneys;

OutputERROR 1222 (21000): The used SELECT statements have a different number of columns

Another way to query multiple tables is through the use of subqueries. Subqueries (also known as inner or nested queries) are queries enclosed within another query. These are useful in cases where you're trying to filter the results of a query against the result of a separate aggregate function.

To illustrate this idea, say you want to know which of your friends have won more matches than Barbara. Rather than querying how many matches Barbara has won then running another query to see who has won more games than that, you can calculate both with a single query:

SELECT name, wins FROM tourneys 
WHERE wins > (
SELECT wins FROM tourneys WHERE name = 'Barbara'
);

Output+--------+------+
| name   | wins |
+--------+------+
| Dolly  |    7 |
| Etta   |    4 |
| Irma   |    9 |
| Gladys |   13 |
+--------+------+
4 rows in set (0.00 sec)

The subquery in this statement was run only once; it only needed to find the value from the wins column in the same row as Barbara in the name column, and the data returned by the subquery and outer query are independent of one another. There are cases, though, where the outer query must first read every row in a table and compare those values against the data returned by the subquery in order to return the desired data. In this case, the subquery is referred to as a correlated subquery.

The following statement is an example of a correlated subquery. This query seeks to find which of your friends have won more games than is the average for those with the same shoe size:

SELECT name, size FROM tourneys AS t 
WHERE wins > (
SELECT AVG(wins) FROM tourneys WHERE size = t.size
);

In order for the query to complete, it must first collect the name and size columns from the outer query. Then, it compares each row from that result set against the results of the inner query, which determines the average number of wins for individuals with identical shoe sizes. Because you only have two friends that have the same shoe size, there can only be one row in the result-set:

Output+------+------+
| name | size |
+------+------+
| Etta |    9 |
+------+------+
1 row in set (0.00 sec)

As mentioned earlier, subqueries can be used to query results from multiple tables. To illustrate this with one final example, say you wanted to throw a surprise dinner for the group's all-time best bowler. You could find which of your friends has the best bowling record and return their favorite meal with the following query:

SELECT name, entree, side, dessert 
FROM dinners 
WHERE name = (SELECT name FROM tourneys 
WHERE wins = (SELECT MAX(wins) FROM tourneys));

Output+--------+--------+-------+-----------+
| name   | entree | side  | dessert   |
+--------+--------+-------+-----------+
| Gladys | steak  | fries | ice cream |
+--------+--------+-------+-----------+
1 row in set (0.00 sec)

Notice that this statement not only includes a subquery, but also contains a subquery within that subquery.

Conclusion

Issuing queries is one of the most commonly-performed tasks within the realm of database management. There are a number of database administration tools, such as phpMyAdmin or pgAdmin, that allow you to perform queries and visualize the results, but issuing SELECT statements from the command line is still a widely-practiced workflow that can also provide you with greater control.

If you're new to working with SQL, we encourage you to use our SQL Cheat Sheet as a reference and to review the official MySQL documentation. Additionally, if you'd like to learn more about SQL and relational databases, the following tutorials may be of interest to you:

An Introduction to Queries in PostgreSQL

2018-10-17T18:50:23Z

Introduction

PostgreSQL, often shortened to "Postgres," is a relational database management system with an object-oriented approach, meaning that information can be represented as objects or classes in PostgreSQL schemas. PostgreSQL aligns closely with standard SQL, although it also includes some features not found in other relational database systems.

Prerequisites

In general, the commands and concepts presented in this guide can be used on any Linux-based operating system running any SQL database software. However, it was written specifically with an Ubuntu 18.04 server running PostgreSQL in mind. To set this up, you will need the following:

An Ubuntu 18.04 machine with a non-root user with sudo privileges. This can be set up using our Initial Server Setup guide for Ubuntu 18.04.
PostgreSQL installed on the machine. For help with setting this up, follow the "Installing PostgreSQL" section of our guide on How To Install and Use PostgreSQL on Ubuntu 18.04.

With this setup in place, we can begin the tutorial.

Creating a Sample Database

For the sample database we'll use throughout this guide, imagine the following scenario:

To begin, open up a PostgreSQL prompt as your postgres superuser:

sudo -u postgres psql

Note: If you followed all the steps of the prerequisite tutorial on Installing PostgreSQL on Ubuntu 18.04, you may have configured a new role for your PostgreSQL installation. In this case, you can connect to the Postgres prompt with the following command, substituting sammy with your own username:

sudo -u sammy psql

Next, create the database by running:

CREATE DATABASE birthdays;

Then select this database by typing:

\c birthdays

CREATE TABLE tourneys ( 
name varchar(30), 
wins real, 
best real, 
size real 
);

Once you run the CREATE TABLE command and populate it with column headings, you’ll receive the following output:

OutputCREATE TABLE

Populate the tourneys table with some sample data:

INSERT INTO tourneys (name, wins, best, size) 
VALUES ('Dolly', '7', '245', '8.5'), 
('Etta', '4', '283', '9'), 
('Irma', '9', '266', '7'), 
('Barbara', '2', '197', '7.5'), 
('Gladys', '13', '273', '8');

You’ll receive the following output:

OutputINSERT 0 5

CREATE TABLE dinners ( 
name varchar(30), 
birthdate date, 
entree varchar(30), 
side varchar(30), 
dessert varchar(30) 
);

Similarly for this table, you’ll receive feedback verifying that the table was created:

OutputCREATE TABLE

Populate this table with some sample data as well:

INSERT INTO dinners (name, birthdate, entree, side, dessert) 
VALUES ('Dolly', '1946-01-19', 'steak', 'salad', 'cake'), 
('Etta', '1938-01-25', 'chicken', 'fries', 'ice cream'), 
('Irma', '1941-02-18', 'tofu', 'fries', 'cake'), 
('Barbara', '1948-12-25', 'tofu', 'salad', 'ice cream'), 
('Gladys', '1944-05-28', 'steak', 'fries', 'ice cream');

OutputINSERT 0 5

Once that command completes successfully, you're done setting up your database. Next, we'll go over the basic command structure of SELECT queries.

Understanding SELECT Statements

Generally, SQL queries follow this syntax:

SELECT column_to_select FROM table_to_select WHERE certain_conditions_apply;

By way of example, the following statement will return the entire name column from the dinners table:

SELECT name FROM dinners;

Output  name   
---------
 Dolly
 Etta
 Irma
 Barbara
 Gladys
(5 rows)

You can select multiple columns from the same table by separating their names with a comma, like this:

SELECT name, birthdate FROM dinners;

Output  name   | birthdate  
---------+------------
 Dolly   | 1946-01-19
 Etta    | 1938-01-25
 Irma    | 1941-02-18
 Barbara | 1948-12-25
 Gladys  | 1944-05-28
(5 rows)

SELECT * FROM tourneys;

Output  name   | wins | best | size 
---------+------+------+------
 Dolly   |    7 |  245 |  8.5
 Etta    |    4 |  283 |    9
 Irma    |    9 |  266 |    7
 Barbara |    2 |  197 |  7.5
 Gladys  |   13 |  273 |    8
(5 rows)

. . . WHERE column_name comparison_operator value

The comparison operator in a WHERE clause defines how the specified column should be compared against the value. Here are some common SQL comparison operators:

Operator	What it does
`=`	tests for equality
`!=`	tests for inequality
`<`	tests for less-than
`>`	tests for greater-than
`<=`	tests for less-than or equal-to
`>=`	tests for greater-than or equal-to
`BETWEEN`	tests whether a value lies within a given range
`IN`	tests whether a row's value is contained in a set of specified values
`EXISTS`	tests whether rows exist, given the specified conditions
`LIKE`	tests whether a value matches a specified string
`IS NULL`	tests for `NULL` values
`IS NOT NULL`	tests for all values other than `NULL`

For example, if you wanted to find Irma's shoe size, you could use the following query:

SELECT size FROM tourneys WHERE name = 'Irma';

Output size 
------
    7
(1 row)

SELECT entree FROM dinners WHERE entree LIKE 't%';

Output entree  
-------
 tofu
 tofu
(2 rows)

Based on the output above, we see that the entree we have forgotten is tofu.

SELECT name AS n, birthdate AS b, dessert AS d FROM dinners;

Output    n    |     b      |     d     
---------+------------+-----------
 Dolly   | 1946-01-19 | cake
 Etta    | 1938-01-25 | ice cream
 Irma    | 1941-02-18 | cake
 Barbara | 1948-12-25 | ice cream
 Gladys  | 1944-05-28 | ice cream
(5 rows)

Here, we have told SQL to display the name column as n, the birthdate column as b, and the dessert column as d.

Aggregate Functions

SELECT COUNT(entree) FROM dinners WHERE entree = 'tofu';

Output count 
-------
     2
(1 row)

The AVG function returns the average (mean) value of a column. Using our example table, you could find the average best score amongst your friends with this query:

SELECT AVG(best) FROM tourneys;

Output  avg  
-------
 252.8
(1 row)

SUM is used to find the total sum of a given column. For instance, if you'd like to see how many games you and your friends have bowled over the years, you could run this query:

SELECT SUM(wins) FROM tourneys;

Output sum 
-----
  35
(1 row)

SELECT SUM(entree) FROM dinners;

OutputERROR:  function sum(character varying) does not exist
LINE 1: select sum(entree) from dinners;
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

MIN is used to find the smallest value within a specified column. You could use this query to see what the worst overall bowling record is so far (in terms of number of wins):

SELECT MIN(wins) FROM tourneys;

Output min 
-----
   2
(1 row)

Similarly, MAX is used to find the largest numeric value in a given column. The following query will show the best overall bowling record:

SELECT MAX(wins) FROM tourneys;

Output max 
-----
  13
(1 row)

SELECT MIN(name) FROM dinners;

Output   min   
---------
 Barbara
(1 row)

Likewise, when run on a column containing string values, the MAX function will show the last value alphabetically:

SELECT MAX(name) FROM dinners;

Output max  
------
 Irma
(1 row)

Manipulating Query Outputs

For example, let's say you wanted to know how many of your friends prefer each of the three entrees you make. You could find this info with the following query:

SELECT COUNT(name), entree FROM dinners GROUP BY entree;

Output count | entree  
-------+---------
     1 | chicken
     2 | steak
     2 | tofu
(3 rows)

SELECT name, birthdate FROM dinners ORDER BY birthdate;

Output  name   | birthdate  
---------+------------
 Etta    | 1938-01-25
 Irma    | 1941-02-18
 Gladys  | 1944-05-28
 Dolly   | 1946-01-19
 Barbara | 1948-12-25
(5 rows)

Notice that the default behavior of ORDER BY is to sort the result-set in ascending order. To reverse this and have the result-set sorted in descending order, close the query with DESC:

SELECT name, birthdate FROM dinners ORDER BY birthdate DESC;

Output  name   | birthdate  
---------+------------
 Barbara | 1948-12-25
 Dolly   | 1946-01-19
 Gladys  | 1944-05-28
 Irma    | 1941-02-18
 Etta    | 1938-01-25
(5 rows)

SELECT COUNT(name), side FROM dinners WHERE COUNT(name) >= 3;

OutputERROR:  aggregate functions are not allowed in WHERE
LINE 1: SELECT COUNT(name), side FROM dinners WHERE COUNT(name) >= 3...

The following example is another attempt to find which side dishes are the favorite of at least three of your friends, although this one will return a result without error:

SELECT COUNT(name), side FROM dinners GROUP BY side HAVING COUNT(name) >= 3;

Output count | side  
-------+-------
     3 | fries
(1 row)

Querying Multiple Tables

More often than not, a database contains multiple tables, each holding different sets of data. SQL provides a few different ways to run a single query on multiple tables.

The JOIN clause can be used to combine rows from two or more tables in a query result. It does this by finding a related column between the tables and sorts the results appropriately in the output.

SELECT statements that include a JOIN clause generally follow this syntax:

SELECT table1.column1, table2.column2
FROM table1
JOIN table2 ON table1.related_column=table2.related_column;

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
JOIN dinners ON tourneys.name=dinners.name;

Output  name   | size | birthdate  
---------+------+------------
 Dolly   |  8.5 | 1946-01-19
 Etta    |    9 | 1938-01-25
 Irma    |    7 | 1941-02-18
 Barbara |  7.5 | 1948-12-25
 Gladys  |    8 | 1944-05-28
(5 rows)

INSERT INTO tourneys (name, wins, best, size) 
VALUES ('Bettye', '0', '193', '9');

INSERT INTO dinners (name, birthdate, entree, side, dessert) 
VALUES ('Lesley', '1946-05-02', 'steak', 'salad', 'ice cream');

Then, re-run the previous SELECT statement with the JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
JOIN dinners ON tourneys.name=dinners.name;

Output  name   | size | birthdate  
---------+------+------------
 Dolly   |  8.5 | 1946-01-19
 Etta    |    9 | 1938-01-25
 Irma    |    7 | 1941-02-18
 Barbara |  7.5 | 1948-12-25
 Gladys  |    8 | 1944-05-28
(5 rows)

Notice that, because the tourneys table has no entry for Lesley and the dinners table has no entry for Bettye, those records are absent from this output.

It is possible, though, to return all the records from one of the tables using an outer JOIN clause. Outer JOIN clauses are written as either LEFT JOIN, RIGHT JOIN, or FULL JOIN.

Run the previous query again, but this time use a LEFT JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
LEFT JOIN dinners ON tourneys.name=dinners.name;

Output  name   | size | birthdate  
---------+------+------------
 Dolly   |  8.5 | 1946-01-19
 Etta    |    9 | 1938-01-25
 Irma    |    7 | 1941-02-18
 Barbara |  7.5 | 1948-12-25
 Gladys  |    8 | 1944-05-28
 Bettye  |    9 | 
(6 rows)

Now run the query again, this time with a RIGHT JOIN clause:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
RIGHT JOIN dinners ON tourneys.name=dinners.name;

Output  name   | size | birthdate  
---------+------+------------
 Dolly   |  8.5 | 1946-01-19
 Etta    |    9 | 1938-01-25
 Irma    |    7 | 1941-02-18
 Barbara |  7.5 | 1948-12-25
 Gladys  |    8 | 1944-05-28
         |      | 1946-05-02
(6 rows)

There is a fourth join clause called FULL JOIN available for some RDBMS distributions, including PostgreSQL. A FULL JOIN will return all the records from each table, including any null values:

SELECT tourneys.name, tourneys.size, dinners.birthdate 
FROM tourneys 
FULL JOIN dinners ON tourneys.name=dinners.name;

Output  name   | size | birthdate  
---------+------+------------
 Dolly   |  8.5 | 1946-01-19
 Etta    |    9 | 1938-01-25
 Irma    |    7 | 1941-02-18
 Barbara |  7.5 | 1948-12-25
 Gladys  |    8 | 1944-05-28
 Bettye  |    9 | 
         |      | 1946-05-02
(7 rows)

Note: As of this writing, the FULL JOIN clause is not supported by either MySQL or MariaDB.

As an alternative to using FULL JOIN to query all the records from multiple tables, you can use the UNION clause.

To illustrate, run the following query:

SELECT name FROM tourneys UNION SELECT name FROM dinners;

This query will remove any duplicate entries, which is the default behavior of the UNION operator:

Output  name   
---------
 Irma
 Etta
 Bettye
 Gladys
 Barbara
 Lesley
 Dolly
(7 rows)

To return all entries (including duplicates) use the UNION ALL operator:

SELECT name FROM tourneys UNION ALL SELECT name FROM dinners;

Output  name   
---------
 Dolly
 Etta
 Irma
 Barbara
 Gladys
 Bettye
 Dolly
 Etta
 Irma
 Barbara
 Gladys
 Lesley
(12 rows)

SELECT name FROM dinners UNION SELECT name, wins FROM tourneys;

OutputERROR:  each UNION query must have the same number of columns
LINE 1: SELECT name FROM dinners UNION SELECT name, wins FROM tourne...

SELECT name, wins FROM tourneys 
WHERE wins > (
SELECT wins FROM tourneys WHERE name = 'Barbara'
);

Output  name  | wins 
--------+------
 Dolly  |    7
 Etta   |    4
 Irma   |    9
 Gladys |   13
(4 rows)

The following statement is an example of a correlated subquery. This query seeks to find which of your friends have won more games than is the average for those with the same shoe size:

SELECT name, size FROM tourneys AS t 
WHERE wins > (
SELECT AVG(wins) FROM tourneys WHERE size = t.size
);

Output name | size 
------+------
 Etta |    9
(1 row)

SELECT name, entree, side, dessert 
FROM dinners 
WHERE name = (SELECT name FROM tourneys 
WHERE wins = (SELECT MAX(wins) FROM tourneys));

Output  name  | entree | side  |  dessert  
--------+--------+-------+-----------
 Gladys | steak  | fries | ice cream
(1 row)

Notice that this statement not only includes a subquery, but also contains a subquery within that subquery.

Conclusion

If you're new to working with SQL, we encourage you to use our SQL Cheat Sheet as a reference and to review the official PostgreSQL documenation. Additionally, if you'd like to learn more about SQL and relational databases, the following tutorials may be of interest to you:

Usando uma CDN para Acelerar a Entrega de Conteúdo Estático

2018-10-17T00:38:31Z

Introdução

Websites e aplicações modernas geralmente entregam uma quantidade significativa de conteúdo estático para os usuários finais. Este conteúdo inclui imagens, folhas de estilo, JavaScript, e vídeo. À medida que esses recursos estáticos aumentam em número e tamanho, o uso da largura de banda aumenta e o tempo de carregamento da página aumenta, deteriorando a experiência de navegação dos usuários e reduzindo a capacidade disponível dos servidores.

Para reduzir drasticamente o tempo de carregamento de página, aumentar a performance e reduzir seus custos com largura de banda e infraestrutura, você pode implementar uma CDN, ou rede de entrega de conteúdo para armazenar em cache esses recursos em um conjunto de servidores distribuídos geograficamente.

Neste tutorial, vamos fornecer uma visão geral de alto nível de CDNs e como elas funcionam, bem como os benefícios que elas podem trazer para suas aplicações web.

O que é uma CDN?

Content delivery network ou rede de entrega de conteúdo é um grupo de servidores geograficamente distribuídos otimizados para entregar conteúdo estático aos usuários finais. Esse conteúdo estático pode ser praticamente qualquer tipo de dados, mas as CDNs são mais comumente usadas para entregar páginas web e seus arquivos relacionados, streaming de vídeo e áudio e grandes pacotes de software.

Uma CDN consiste em múltiplos pontos de presença (PoPs) em várias localidades, cada qual consistindo de vários servidores de borda que armazenam em cache recursos de sua origem, ou servidor de hospedagem. Quando um usuário visita seu website e solicita recursos estáticos como iamgens ou arquivos de JavaScript, suas solicitações são encaminhadas pela CDN para o servidor de borda mais próximo, a partir do qual o conteúdo é servido. Se o servidor de borda não tem os recursos em cache ou o cache de recursos expirou, a CDN irá buscar e armazenar em cache a versão mais recente de outro servidor de borda da CDN mais próximo ou de seus servidores de origem. Se a borda da CDN tiver uma entrada de cache para seus recursos (o que ocorre na maior parte do tempo se seu site receber uma quantidade moderada de tráfego), ela retornará a cópia em cache para o usuário final.

Isso permite que usuários geograficamente dispersos minimizem o número de saltos necessários para receber o conteúdo estático, buscando o conteúdo diretamente do cache de uma borda próxima. O resultado é latência e perda de pacotes significativamente reduzidos, tempos de carregamento de página mais rápidos e uma carga drasticamente reduzida na sua infraestrutura de origem.

Os provedores de CDN oferecem recursos adicionais como mitigação de DDoS e limitação de taxa, análise de usuários e otimizações para casos de uso móvel ou de streaming com custo adicional.

Como uma CDN funciona?

Quando um usuário visita seu website, ele primeiro recebe uma resposta de um servidor de DNS contendo o endereço IP do host do seu servidor web. Seu navegador então solicita o conteúdo da página web, que geralmente consite de uma variedade de arquivos estáticos, como páginas HTML, folhas de estilo CSS, código JavaScript e imagens.

Uma vez lançada a CDN e descarregados esses recursos estáticos em sevidores da CDN, "empurrando-os" manualmente ou fazendo com que a CDN "puxe" os recursos automaticamente (ambos os mecanismos são cobertos na próxima seção), você então instrui seu webserver a reescrever os links para conteúdo estático, de modo que esses links agora apontem para arquivos hospedados pela CDN. Se você estiver usando um CMS como o WordPress, essa reescrita de link pode ser implementada usando um plugin de terceiros como o CDN Enabler.

Muitas CDNs fornecem suporte para domínios personalizados, permitindo que você crie um registro CNAME em seu domínio apontando para um endpoint da CDN. Depois que a CDN recebe uma solicitação do usuário nesse endpoint (localizado na borda, muito mais perto do usuário do que seus servidores de back-end), ela então encaminha a solicitação para o Ponto de Presença (PoP) localizado mais próximo do usuário. Este PoP geralmente consiste de um ou mais servidores de borda da CDN colocados em um ponto de troca de Internet (IxP), essencialmente um datacenter que os Provedores de Serviço de Internet (ISPs) utilizam para interconectar suas redes. O balanceador de carga interno da CDN então encaminha a solicitação para um servidor de borda localizado neste PoP, que então serve o conteúdo para o usuário.

Os mecanismos de cache variam entre os provedores de CDN, mas geralmente funcionam da seguinte maneira:

Quando a CDN recebe uma primeira solicitação para um recurso estático, como uma imagem PNG, ele não tem o recurso em cache e deve buscar uma cópia do recurso de um servidor de borda de CDN próximo ou do próprio servidor de origem. Isso é conhecido como cache "miss" e geralmente pode ser detectado inspecionando o cabeçalho de resposta HTTP, contendo X-Cache: MISS. Essa solicitação inicial será mais lenta que as solicitações futuras porque, depois de concluir essa solicitação, o recurso terá sido armazenado em cache na borda.
As solicitações futuras para esse recurso ("hits" do cache), encaminhadas para esse local de borda, agora serão atendidas a partir do cache, até a expiração (normalmente definida através de cabeçalhos HTTP). Essas respostas serão significativamente mais rápidas do que a solicitação inicial, reduzindo drasticamente as latências para os usuários e transferindo o tráfego web para a rede da CDN. Você pode verificar se a resposta foi atendida a partir de um cache CDN, inspecionando o cabeçalho de resposta HTTP, que agora deve conter X-Cache: HIT.

Para saber mais sobre como uma CDN específica funciona e como foi implementada, consulte a documentação do seu provedor de CDN.

Na próxima seção, vamos introduzir os dois tipos populares de CDNs: As CDNs push e pull

Zonas Push versus Zonas Pull

A maioria dos provedores de CDN oferecem duas maneiras de armazenar seus dados em cache: zonas pull e zonas push.

Zonas Pull envolvem a entrada do endereço do seu servidor de origem, e deixar a CDN buscar e armazenar em cache automaticamente todos os recursos estáticos disponíveis em seu site. Zonas Pull são comumente usadas para fornecer recursos web de pequeno a médio porte frequentemente atualizados, como arquivos HTML, CSS e arquivos JavaScript. Depois de fornecer à CDN o endereço do servidor de origem, a próxima etapa é geralmente reconfigurar links para recursos estáticos, de forma que eles agora apontem para o URL fornecido pela CDN. A partir desse ponto, o CDN manipulará as solicitações de recursos de entrada dos usuários e fornecerá conteúdo de seus caches geograficamente distribuídos e sua origem, conforme apropriado.

Para utilizar uma Zona Push, você faz o upload de seus dados para um bucket ou local de armazenamento designado, que a CDN então envia para caches em sua frota distribuída de servidores de borda. As zonas Push são normalmente usadas para arquivos maiores, atualizados com pouca frequência, como arquivamentos, pacotes de software, PDFs, vídeo e arquivos de áudio.

Benefícios do Uso de uma CDN

Quase qualquer site pode colher os benefícios fornecidos pela implementação de uma CDN, mas geralmente as principais razões para implementá-la são descarregar a largura de banda de seus servidores de origem nos servidores CDN e reduzir a latência para usuários distribuídos geograficamente.

Vamos passar por estas e várias outras grandes vantagens oferecidas pelo uso de uma CDN logo abaixo.

Descarregamento da Origem

Se você estiver se aproximando da capacidade de largura de banda em seus servidores, o descarregamento de recursos estáticos como imagens, vídeos, arquivos CSS e JavaScript reduzirá drasticamente o uso da largura de banda dos servidores. Redes de entrega de conteúdo são projetadas e otimizadas para servir conteúdo estático, e as solicitações de clientes para esse conteúdo serão encaminhadas e servidas por servidores CDN de borda. Isso traz o benefício adicional de reduzir a carga em seus servidores de origem, pois eles servem esses dados com uma frequência muito menor.

Latência Mais Baixa para uma Melhor Experiência do Usuário

Se sua base de usuários estiver geograficamente dispersa e uma parte não trivial de seu tráfego vier de uma área geográfica distante, uma CDN poderá diminuir a latência ao colocar em cache os recursos estáticos em servidores de borda mais próximos dos seus usuários. Ao reduzir a distância entre os usuários e o conteúdo estático, você pode fornecer conteúdo para seus usuários com mais rapidez e melhorar a experiência aumentando as velocidades de carregamento de páginas.

Esses benefícios são compostos para websites que atendem principalmente a conteúdo de vídeo com uso intensivo de largura de banda, em que altas latências e lentidão no tempo de carregamento afetam diretamente a experiência do usuário e o engajamento de conteúdo.

Gerenciar Picos de Tráfego e Evitar Tempo de Inatividade

As CDNs permitem lidar com grandes picos e rajadas de tráfego através do balanceamento da carga de solicitações em uma grande rede distribuída de servidores de borda. Ao descarregar e armazenar em cache o conteúdo estático em uma rede de entrega, você pode acomodar um número maior de usuários simultâneos com sua infraestrutura atual.

Para sites que usam um único servidor de origem, esses grandes picos de tráfego podem sobrecarregar o sistema, causando interrupções e tempo de inatividade não planejados. A transferência do tráfego para uma infraestrutura de CDN altamente disponível e redundante, projetada para lidar com níveis variáveis de tráfego web, pode aumentar a disponibilidade de seus recursos e do seu conteúdo.

Reduzir Custos

Como a veiculação de conteúdo estático geralmente ocupa a maior parte do uso da largura de banda, o descarregamento desses recursos em uma rede de distribuição de conteúdo pode reduzir drasticamente os gastos mensais com infraestrutura. Além de reduzir os custos de largura de banda, uma CDN pode reduzir os custos de servidor através da redução da carga nos servidores de origem, permitindo escalar a infraestrutura existente. Por fim, alguns provedores de CDN oferecem faturamento mensal com preço fixo, permitindo que você transforme seu uso de largura de banda mensal variável em um gasto recorrente estável e previsível.

Aumentar a Segurança

Outro caso de uso comum para CDNs é a mitigação de ataques de DDoS. Muitos provedores de CDN incluem recursos para monitorar e filtrar solicitações para servidores de borda. Esses serviços analisam o tráfego web em busca de padrões suspeitos, bloqueando o tráfego de ataque mal-intencionado e, ao mesmo tempo, permitindo o tráfego de usuários confiáveis. Os provedores de CDN geralmente oferecem uma variedade de serviços de mitigação de DDoS, desde proteção contra ataques comuns no nível de infra-estrutura (camadas OSI 3 e 4), até serviços de mitigação mais avançados e limitação de taxa.

Além disso, a maioria das CDNs permite configurar o SSL completo, para que você possa criptografar o tráfego entre a CDN e o usuário final, bem como o tráfego entre a CDN e seus servidores de origem, usando certificados SSL personalizados ou fornecidos pela própria CDN.

Escolhendo a Melhor Solução

Se o seu gargalo for a carga da CPU no servidor de origem e não a largura de banda, uma CDN pode não ser a solução mais apropriada. Nesse caso, o cache local usando caches populares, como NGINX ou Varnish, pode reduzir significativamente a carga servindo os recursos a partir da memória do sistema.

Antes de lançar uma CDN, etapas adicionais de otimização — como minimizar e compactar arquivos JavaScript e CSS e ativar a compactação de solicitação HTTP do servidor Web — podem também ter um impacto significativo em tempos de carregamento de páginas e uso de largura de banda.

Uma ferramenta útil para avaliar a velocidade de carregamento de página e melhorá-la é o PageSpeed Insights do Google. Outra ferramenta útil que fornece um detalhamento em cascata de solicitações e tempos de resposta, bem como otimizações sugeridas é o Pingdom.

Conclusão

Uma rede de entrega de conteúdo pode ser uma solução rápida e eficaz para melhorar a escalabilidade e disponibilidade de seus websites. Ao armazenar em cache os recursos estáticos em uma rede geograficamente distribuída de servidores otimizados, você pode reduzir muito os tempos de carregamento de página e as latências para os usuários finais. Além disso, as CDNs permitem que você reduza significativamente o uso de largura de banda, absorvendo as solicitações dos usuários e respondendo a partir do cache na borda, reduzindo assim seus custos de largura de banda e infra-estrutura.

Com plugins e suporte de terceiros para os principais frameworks como WordPress, Drupal, Django e Ruby on Rails, além de recursos adicionais como mitigação de DDoS, SSL completo, monitoramento de usuários e compactação de recursos, as CDNs podem ser uma ferramenta de impacto para proteger e otimizar websites de alto tráfego.

Por Hanif Jetha

Como criar um Space e uma API Key na DigitalOcean

2018-10-11T20:59:03Z

Introdução

DigitalOcean Spaces é um serviço de armazenamento de objetos que torna mais fácil e econômico armazenar e fornecer grandes quantidades de dados. Spaces individuais podem ser criados e colocados em uso rapidamete, sem necessidade de configuração adicional.

Neste tutorial, iremos utilizar o Painel de Controle da DigitalOcean para criar um novo Space. Em seguida, iremos recuperar uma Chave de API ou API Key e um secret que podem ser utilizados para conceder acesso ao Space para quaisquer clientes ou bibliotecas compatíveis com S3.

Pré-requisitos

Para completar este tutorial, você vai precisar de uma conta na DigitalOcean. Se você já não tiver uma, você pode registrar uma nova na página de inscrição.

Faça o login no Painel de Controle da DigitalOcean para começar.

Criando um Space

Para criar um novo Space, utilize o botão Create no canto superior direito do Painel de Controle. Clique no botão, em seguida escolha Spaces na lista suspensa:

Se você nunca criou um Space antes, você também pode criar um diretamente da página do Spaces. Para fazer isso, clique Spaces na navegação principal do Painel de Controle, e então clique em Create a space. Qualquer uma das opções o levarão à tela Create a Space:

Primeiro, escolha um nome para o seu Space. Esse nome deve ser único entre todos os Spaces (ou seja, nenhum outro usuário do Spaces pode ter o mesmo nome em qualquer região), deve ter de 3 a 63 caracteres, e pode conter apenas letras minúsculas, números e traços.

Em seguida, escolha a região do datacenter onde você gostaria que seu Space estivesse. No momento em que esta captura de tela foi feita, nyc3 e ams3 eram as escolhas possíveis. Outras se tornarão dsponíveis ao longo do tempo.

Finalmente, escolha se deseja que os usuários não autenticados possam listar todos os arquivos em seu Space. Isso não afeta o acesso a arquivos individuais (que é definido em uma base por aquivo), mas apenas a capacidade de obter uma lista de todos os arquivos. A escolha padrão de Private é segura, a menos que você tenha alguns scripts ou clientes que precisem buscar listagens de arquivos sem uma chave de acesso ou access key.

Quando seu nome e as opções estiverem todos definidos, desça e clique no botão Create a Space. Seu Space será criado, e você será levado para a interface do navegador de arquivos:

Se este é o seu primeiro Space, você terá um arquivo welcome.html, do contrário, o Space estará vazio.

Tome nota da URL do seu Space. Está disponível logo abaixo do nome do Space na visualização do navegador de arquivos. Nesse caso de exemplo, a URL completa é https://example-name.nyc3.digitaloceanspaces.com. O nome do Space aqui (geralmente chamado de nome do bucket) é example-name. A URL do servidor (ou endereço) é a parte restante, consistindo do nome do datacenter seguido por .digitaloceanspaces.com: https://nyc3.digitaloceanspaces.com.

Existem algumas maneiras diferentes pelas quais os clientes e bibliotecas solicitarão essas informações. Alguns vão querer no mesmo formato dado no Painel de Controle. Alguns exigem que o nome do bucket siga a URL do servidor, como em https://nyc3.digitaloceanspaces.com/example-name. Outros ainda pedirão para você inserir o endereço do servidor e o nome do bucket ou Space separadamente. Consulte a documentação do seu cliente ou biblioteca para mais orientações nesse item.

A seguir, criaremos a chave que precisamos para acessar nossos Spaces a partir de clientes de terceiros.

Criando uma Access Key

Para acessar seus arquivos de fora do Painel de Controle da DigitalOcean, precisamos gerar uma chave de acesso ou access key e um secret. Estes são um par de tokens aleatórios que servem como nome de usuário e senha para conceder acesso ao seu Space.

Primeiro, clique no link da API na navegação principal do Painel de Controle. A página resultante lista seus tokens de API da DigitalOcean e as chaves de acesso do Spaces. Role para baixo até a parte do Spaces:

Se este é o seu primeiro Space, você não pode ter nenhuma chave listada. Clique no botão Generate New Key. A caixa de diálogo New Spaces key será exibida:

Digite um nome para a chave. Você pode criar quantas chaves quiser, portanto, lembre-se de que a única maneira de revogar o acesso a uma chave é excluí-la. Desse modo, você pode querer particionar as chaves por pessoa, por equipe ou pelo software cliente no qual você as estiver utilizando.

Neste caso, estamos criando uma chave chamada example-token. Clique no botão Generate Key para completar o processo. Você retornará à tela da API listando todas as suas chaves. Observe que a nova chave tem dois tokens longos exibidos:

O primeiro é a sua access key. Isso não é secreto e continuará visível no Painel de Controle. A segunda string é o seu secret ou secret key. Isso só será exibido uma vez. Registre-a em um local seguro para uso posterior. Na próxima vez que você visitar a página da API, esse valor será eliminado e não há como recuperá-lo.

Diferentes clientes compatíveis com S3 podem ter nomes sutilmente diferentes para access key e secret. A terminologia usada é normalmente próxima o suficiente para deixar claro qual token deve ir para onde. Caso contrário, consulte a documentação do seu cliente ou biblioteca para obter mais informações.

Conclusão

Neste tutorial criamos um novo Space na DigitalOcean e uma nova access key e secret. Agora sabemos nossa URL de servidor, nome do bucket (ou nome do Space), access key, e secret. Com essas informações você pode conectar praticamente qualquer cliente ou biblioteca compatível com S3 ao seu novo Space na DigitalOcean!

Por Brian Boucheron

Como Configurar Pipelines de Integração Contínua com o GitLab CI no Ubuntu 16.04

2018-10-11T03:28:49Z

Introdução

O GitLab Community Edition é um provedor de repositório Git auto-hospedado com recursos adicionais para ajudar no gerenciamento de projetos e no desenvolvimento de software. Um dos recursos mais valiosos que o GitLab oferece é a ferramenta embutida de integração e entrega contínua chamada GitLab CI.

Neste guia, vamos demonstrar como configurar o GitLab CI para monitorar seus repositórios por mudanças e executar testes automatizados para validar código novo. Começaremos com uma instalação do GitLab em execução, na qual copiaremos um repositório de exemplo para uma aplicação básica em Node.js. Depois de configurarmos nosso processo de CI, quando um novo commit é enviado ao repositório o GitLab irá utilizar o CI runner para executar o conjunto de testes em cima do código em um container Docker isolado.

Pré-requisitos

Antes de começarmos, você precisará configurar um ambiente inicial. Vamos precisar de um servidor GitLab seguro configurado para armazenar nosso código e gerenciar nosso processo de CI/CD. Adicionalmente, precisaremos de um local para executar os testes automatizados. Este pode ser o mesmo servidor em que o GitLab está instalado ou um host separado. As seções abaixo cobrem os requisitos em mais detalhes.

Um Servidor GitLab Protegido com SSL

Para armazenar nosso código-fonte e configurar nossas tarefas de CI/CD, precisamos de uma instância do GitLab instalada em um servidor Ubuntu 16.04. Atualmente o GitLab recomenda um servidor com no mínimo 2 núcleos de CPU e 4GB de RAM. Para proteger seu código de ser exposto ou adulterado, a instância do GitLab será protegida com SSL usando o Let's Encrypt. Seu servidor precisa ter um nome de domínio associado a ele para completar essa etapa.

Você pode atender esses requisitos usando os seguintes tutoriais:

Configuração Inicial de servidor com Ubuntu 16.04: Crie um usuário com privilégios sudo e configure um firewall básico.
Como Instalar e Configurar o GitLab no Ubuntu 16.04: Instale o GitLab no servidor e proteja-o com um certificado Let's Encrypt TLS/SSL.

Estaremos demonstrando como compartilhar CI/CD runners (os componentes que executam os testes automatizados). Se você deseja compartilhar CI runners entre projetos, recomendamos fortemente que você restrinja ou desative as inscrições públicas. Se você não modificou suas configurações durante a instalação, volte e siga a etapa opcional do artigo de instalação do GitLab sobre como restringir ou desabilitar as inscrições para evitar abusos por parte de terceiros.

Um ou Mais Servidores para Utilizar como GitLab CI Runners

GitLab CI Runners são os servidores que verificam o código e executam testes automatizados para validar novas alterações. Para isolar o ambiente de testes, estaremos executando todos os nossos testes automatizados em containers Docker. Para fazer isso, precisamos instalar o Docker no servidor ou servidores que irão executar os testes.

Esta etapa pode ser concluída no servidor GitLab ou em outro servidor Ubuntu 16.04 para fornecer isolamento adicional e evitar contenção de recursos. Os seguintes tutoriais instalarão o Docker no host que você deseja usar para executar seus testes:

Configuração Inicial de servidor com Ubuntu 16.04: Crie um usuário com privilégios sudo e configure um firewall básico. (você não precisa completar isso novamente se estiver configurando o CI runner no servidor do GitLab).
Como Instalar e Usar o Docker no Ubuntu 16.04: Siga os passos 1 e 2 para instalar o Docker no servidor.

Quando estiver pronto para começar, continue com este guia.

Copiando o Repositório de Exemplo a partir do GitHub

Para começar, vamos criar um novo projeto no GitLab contendo a aplicação de exemplo em Node.js. Iremos importar o repositório original diretamente do GitHub para que não tenhamos que carregá-lo manualmente.

Efetue o login no GitLab e clique no ícone de adição no canto superior direito e selecione New project para adicionar um novo projeto:

Na página do novo projeto, clique na aba Import project:

A seguir, clique no botão Repo by URL. Embora exista uma opção de importação do GitHub, ela requer um token de acesso Pessoal e é usada para importar o repositório e informações adicionais. Estamos interessados apenas no código e no histórico do Git, portanto, importar pela URL é mais fácil.

No campo Git repository URL, insira a seguinte URL do repositório GitHub:

https://github.com/do-community/hello_hapi.git

Deve se parecer com isto:

Como esta é uma demonstração, provavelmente é melhor manter o repositório marcado como Private ou privado. Quando terminar, clique em Create project.

O novo projeto será criado baseado no repositório importado do Github.

Entendendo o arquivo .gitlab-ci.yml

O GitLab CI procura por um arquivo chamado .gitlab-ci.yml dentro de cada repositório para determinar como ele deve testar o código. O repositório que importamos já tem um arquivo .gitlab-ci.yml configurado para o projeto. Você pode aprender mais sobre o formato lendo a documentação de referência do .gitlab-ci.yml.

Clique no arquivo .gitlab-ci.yml na interface do GitLab para o projeto que acabamos de criar. A configuração do CI deve ser algo assim:

.gitlab-ci.yml


image: node:latest

stages:
  - build
  - test

cache:
  paths:
    - node_modules/

install_dependencies:
  stage: build
  script:
    - npm install
  artifacts:
    paths:
      - node_modules/

test_with_lab:
  stage: test
  script: npm test

O arquivo utiliza a sintaxe de configuração YAML no GitLab CI para definir as ações que devem ser tomadas, a ordem na qual elas devem executar, sob quais condições elas devem ser executadas e os recursos necessários para concluir cada tarefa. Ao escrever seus próprios arquivos de CI do GitLab, você pode checar com um validador indo até /ci/lint em sua instância GitLab para validar que seu arquivo está formatado corretamente.

O arquivo de configuração começa declarando uma image ou imagem do Docker que deve ser usada para executar o conjunto de testes. Como o Hapi é um framework Node.js, estamos usando a imagem Node.js mais recente:

image: node:latest

Em seguida, definimos explicitamente os diferentes estágios de integração contínua que serão executados:

stages:
  - build
  - test

Os nomes que você escolhe aqui são arbitrários, mas a ordenação determina a ordem de execução dos passos que se seguirão. Stages ou estágios são tags que você pode aplicar a jobs individuais. O GitLab vai executar jobs do mesmo estágio em paralelo e vai esperar para executar o próximo estágio até que todos os jobs do estágio atual estejam completos. Se nenhum estágio for definido, o GitLab usará três estágios chamados build, test, e deploy e atribuir todos os jobs ao estágio test por padrão.

Após definir os estágios, a configuração inclui uma definição de cache:

cache:
  paths:
    - node_modules/

Isso especifica arquivos ou diretórios que podem ser armazenados em cache (salvos para uso posterior) entre execuções ou estágios. Isso pode ajudar a diminuir o tempo necessário para executar tarefas que dependem de recursos que podem não ser alterados entre execuções. Aqui, estamos fazendo cache do diretório node_modules, que é onde o npm instala as dependências que ele baixa.

Nosso primeiro job é chamado install_dependencies:

install_dependencies:
  stage: build
  script:
    - npm install
  artifacts:
    paths:
      - node_modules/

Os jobs podem ter qualquer nome, mas como os nomes serão usados na interface do usuário do GitLab, nomes descritivos são úteis. Normalmente, o npm install pode ser combinado com os próximos estágios de teste, mas para melhor demonstrar a interação entre os estágios, estamos extraindo essa etapa para executar em seu próprio estágio.

Marcamos o estágio explicitamente como "build" com a diretiva stage. Em seguida, especificamos os comandos reais a serem executados usando a diretiva script. Você pode incluir vários comandos inserindo linhas adicionais dentro da seção script.

A sub-seção artifacts é utilizada para especificar caminhos de arquivo ou diretório para salvar e passar entre os estágios. Como o comando npm install instala as dependências do projeto, nossa próxima etapa precisará de acesso aos arquivos baixados. A declaração do caminho node_modules garante que o próximo estágio terá acesso aos arquivos. Estes estarão também disponíveis para visualizar ou baixar na interface de usuário do GitLab após o teste, assim isso é útil para construir artefatos como binários também. Se você quiser salvar tudo que foi produzido durante o estágio, substitua a seção path inteira por untracked: true.

Finalmente, o segundo job chamado test_with_lab declara o comando que realmente executará o conjunto de testes:

test_with_lab:
  stage: test
  script: npm test

Colocamos isso no estágio test. Como esse é o último estágio, ele tem acesso aos artefatos produzidos pelo estágio build que são as dependências do projeto em nosso caso. Aqui, a seção script demonstra a sintaxe YAML de linha única que pode ser usada quando há apenas um único item. Poderíamos ter usado essa mesma sintaxe no job anterior, já que apenas um comando foi especificado.

Agora que você tem uma ideia básica sobre como o arquivo .gitlab-ci.yml define tarefas CI/CD, podemos definir um ou mais runners capazes de executar o plano de testes.

Disparando uma Execução de Integração Contínua

Como o nosso repositório inclui um arquivo .gitlab-ci.yml, quaisquer novos commits irão disparar uma nova execução de CI. Se não houver runners disponíveis, a execução da CI será definida como "pending" ou pendente. Antes de definirmos um runner, vamos disparar uma execução de CI para ver como é um job no estado pendente. Uma vez que um runner esteja disponível, ele imediatamente pegará a execução pendente.

De volta à visão do repositório do projeto do GitLab hello_hapi, clique no sinal de adição ao lado do branch e do nome do projeto e selecione New file no menu:

Na próxima página, insira dummy_file no campo File name e insira algum texto na janela principal de edição:

Clique em commit changes na parte inferior quando terminar.

Agora, retorne à página principal do projeto. Um pequeno ícone de pausa será anexado ao commit mais recente. Se você passar o mouse sobre o ícone, ele irá exibir "Commit:pending":

Isso significa que os testes que validam as alterações de código ainda não foram executados.

Para obter mais informações, vá para o topo da página e clique em Pipelines. Você será direcionado para a página de visão geral do pipeline, na qual é possível ver que a execução CI está marcada como pending e rotulada como "stuck":

Nota: Do lado direito há um botão para a ferramenta CI Lint. É aqui que você pode verificar a sintaxe de qualquer arquivo gitlab-ci.yml que você escreve.

A partir daqui, você pode clicar no status pending para obter mais detalhes sobre a execução. Esta visão mostra os diferentes estágios de nossa execução, bem como os jobs individuais associados a cada estágio:

Finalmente, clique no job install_dependencies. Isso lhe dará detalhes específicos sobre o que está atrasando a execução:

Aqui, a mensagem indica que o trabalho está preso devido à falta de runners. Isso é esperado, uma vez que ainda não configuramos nenhum. Quando um runner estiver disponível, essa mesma interface poderá ser usada para ver a saída. Este é também o local onde você pode baixar os artefatos produzidos durante o build.

Agora que sabemos como é um job pendente, podemos atribuir um runner de CI ao nosso projeto para pegar o job pendente.

Instalando o Serviço CI Runner do GitLab

Agora estamos prontos para configurar um CI Runner do GitLab. Para fazer isso, precisamos instalar o pacote CI runner do GitLab no sistema e iniciar o serviço do runner. O serviço pode executar várias instâncias do runner para projetos diferentes.

Como mencionado nos pré-requisitos, você pode completar estes passos no mesmo servidor que hospeda sua instância do GitLab ou em um servidor diferente se você quiser ter certeza de evitar a contenção de recursos. Lembre-se de que, seja qual for o host escolhido, você precisa do Docker instalado para a configuração que usaremos.

O processo de instalação do serviço CI runner do GitLab é similar ao processo usado para instalar o próprio GitLab. Iremos baixar um script para adicionar um repositório GitLab à nossa lista de fontes apt. Depois de executar o script, faremos o download do pacote do runner. Podemos então configurá-lo para servir nossa instância do GitLab.

Comece baixando a versão mais recente do script de configuração do repositório do GitLab CI runner para o diretório /tmp (este é um repositório diferente daquele usado pelo servidor GitLab):

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh -o /tmp/gl-runner.deb.sh

Sinta-se à vontade para examinar o script baixado para garantir que você está confortável com as ações que ele irá tomar. Você também pode encontrar uma versão hospedada do script aqui:

less /tmp/gl-runner.deb.sh

Quando estiver satisfeito com a segurança do script, execute o instalador:

sudo bash /tmp/gl-runner.deb.sh

O script irá configurar seu servidor para usar os repositórios mantidos pelo GitLab. Isso permite gerenciar os pacotes do runner do GitLab com as mesmas ferramentas de gerenciamento de pacotes que você usa para os outros pacotes do sistema. Quando isso estiver concluído, você pode prosseguir com a instalação usando apt-get:

sudo apt-get install gitlab-runner

Isso irá instalar o pacote CI runner do GitLab no sistema e iniciar o serviço GitLab runner.

Configurando um GitLab Runner

Em seguida, precisamos configurar um CI runner do GitLab para que ele possa começar a aceitar trabalho.

Para fazer isso, precisamos de um token do GitLab runner para que o runner possa se autenticar com o servidor GitLab. O tipo de token que precisamos depende de como queremos usar esse runner.

Um runner específico do projeto é útil se você tiver requisitos específicos para o runner. Por exemplo, se seu arquivo gitlab-ci.yml define tarefas de deployment que requeiram credenciais, um runner específico pode ser necessário para autenticar corretamente dentro do ambiente de deployment. Se o seu projeto tiver etapas com recursos intensivos no processo do CI, isso também pode ser uma boa ideia. Um runner específico do projeto não irá aceitar jobs de outros projetos.

Por outro lado, um runner compartilhado é um runner de propósito geral que pode ser utilizado por vários projetos. Os runners receberão jobs dos projetos de acordo com um algoritmo que contabiliza o número de jobs que estão sendo executados atualmente para cada projeto. Esse tipo de runner é mais flexível. Você precisará fazer login no GitLab com uma conta de administrador para configurar os runners compartilhados.

Vamos demonstrar como obter os tokens de runner para esses dois tipos de runner abaixo. Escolha o método que melhor lhe convier.

Coletando Informações para Registrar um Runner Específico de Projeto

Se você quiser que o runner seja vinculado a um projeto específico, comece navegando até a página do projeto na interface do GitLab.

A partir daqui, clique no item Settings no menu à esquerda. Depois, clique no item CI/CD no submenu:

Nesta página, você verá uma seção Runners settings. Clique no botão Expand para ver mais detalhes. Na visão de detalhes, o lado esquerdo explicará como registrar um runner específico do projeto. Copie o token de registro exibido na etapa 4 das instruções:

Se você quiser desativar quaisquer runners compartilhados ativos para este projeto, você pode fazê-lo clicando no botão Disable shared Runners no lado direito. Isso é opcional.

Quando estiver pronto, avance para aprender como registrar seu runner usando as informações coletadas nesta página.

Coletando Informações para Registrar um Runner Compartilhado

Para encontrar as informações necessárias para registrar um runner compartilhado, você precisa estar logado com uma conta administrativa.

Comece clicando no ícone de chave inglesa na barra de navegação superior para acessar a área administrativa. Na seção Overview do menu à esquerda, clique em Runners para acessar a página de configuração do runner compartilhado.

Copie o token de registro exibido na parte superior da página:

Usaremos esse token para registrar um runner do GitLab CI para o projeto.

Registrando um Runner do GitLab CI com o Servidor GitLab

Agora que você tem um token, volte para o servidor em que seu serviço do runner do GitLab CI está instalado.

Para registrar um novo runner, digite o seguinte comando:

sudo gitlab-runner register

Você será solicitado a responder uma série de questões para configurar o runner:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/)

Insira o nome de domínio do seu servidor GitLab, usando https:// para especificar SSL. Você pode, opcionalmente, anexar /ci ao final do seu domínio, mas as versões recentes serão redirecionadas automaticamente.

Please enter the gitlab-ci token for this runner

Insira o token que você copiou na última seção.

Please enter the gitlab-ci description for this runner

Insira um nome para esse runner particular. Isso será exibido na lista de runners do serviço, na linha de comando e na interface do GitLab.

Please enter the gitlab-ci tags for this runner (comma separated)

Estas são tags que você pode atribuir ao runner. Os jobs do GitLab podem expressar requisitos em termos dessas tags para garantir que eles sejam executados em um host com as dependências corretas.

Você pode deixar isso em branco neste caso.

Whether to lock Runner to current project [true/false]

Atribua o runner ao projeto específico. Ele não poderá ser utilizado por outro projeto.

Selecione "false" aqui.

Please enter the executor

Insira o método usado pelo runner para completar jobs.

Escolha "docker" aqui.

Please enter the default Docker image (e.g. ruby:2.1)

Insira a imagem padrão utilizada para executar jobs quando o arquivo .gitlab-ci.yml não incluir uma especificação de imagem. É melhor especificar uma imagem geral aqui e definir imagens mais específicas em seu arquivo .gitlab-ci.yml como fizemos.

Vamos inserir "alpine:latest" aqui como um padrão pequeno e seguro.

Depois de responder às questões, um novo runner será criado, capaz de executar os jobs de CI/CD do seu projeto.

Você pode ver os runners que o serviço de runner do GitLab CI tem atualmente disponíveis digitando:

sudo gitlab-runner list

OutputListing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
example-runner                                      Executor=docker Token=e746250e282d197baa83c67eda2c0b URL=https://example.com

Agora que temos um runner disponível, podemos retornar ao projeto no GitLab.

Visualizando a Execução de CI/CD no GitLab

De volta ao seu navegador, retorne ao seu projeto no GitLab. Dependendo de quanto tempo passou desde o registro do seu runner, ele pode estar em execução no momento:

Ou ele já pode ter sido concluído:

Independentemente do estado, clique no ícone running ou passed (ou failed se você se deparou com um problema) para ver o estado atual da execução da CI. Você pode ter uma visualização semelhante clicando no menu superior Pipelines.

Você será direcionado para a página de visão geral do pipeline, na qual poderá ver o status da execução do GitLab CI:

No cabeçalho Stages, haverá um círculo indicando o status de cada um dos estágios da execução. Se você clicar no estágio, poderá ver os jobs individuais associados ao estágio:

Clique no job install_dependencies dentro do estágio build. Isso o levará para a página de visão geral do job:

Agora, em vez de exibir uma mensagem de nenhum runner estar disponível, a saída do job é exibida. Em nosso caso, isso significa que você pode ver os resultados do npm instalando cada um dos pacotes.

Ao longo do lado direito, você pode ver alguns outros itens também. Você pode ver outros jobs alterando o estágio e clicando nas execuções abaixo. Você também pode visualizar ou baixar quaisquer artefatos produzidos pela execução.

Conclusão

Neste guia, adicionamos um projeto demonstrativo à instância do Gitlab para mostrar os recursos de integração contínua e de deployment do GitLab CI. Discutimos como definir um pipeline nos arquivos gitlab-ci.yml para construir e testar suas aplicações e como atribuir jobs aos estágios para definir a relação um com o outro. Em seguida, configuramos um runner do GitLab CI para pegar tarefas de CI para nosso projeto e demonstramos como encontrar informações sobre execuções individuais da CI do GitLab.

Por Justin Ellingwood

How To Use Git: A Reference Guide

2018-10-04T20:49:46Z

Git Cheat Sheet

Introduction

Teams of developers and open-source software maintainers typically manage their projects through Git, a distributed version control system that supports collaboration.

This cheat sheet-style guide provides a quick reference to commands that are useful for working and collaborating in a Git repository. To install and configure Git, be sure to read “How To Contribute to Open Source: Getting Started with Git.”

How to Use This Guide:

This guide is in cheat sheet format with self-contained command-line snippets.
Jump to any section that is relevant to the task you are trying to complete.
When you see highlighted text in this guide’s commands, keep in mind that this text should refer to the commits and files in your own repository.

Set Up and Initialization

Check your Git version with the following command, which will also confirm that Git is installed.

git --version

You can initialize your current working directory as a Git repository with init.

git init

To copy an existing Git repository hosted remotely, you’ll use git clone with the repo’s URL or server location (in the latter case you will use ssh).

git clone https://www.github.com/username/repo-name

Show your current Git directory’s remote repository.

git remote

For a more verbose output, use the -v flag.

git remote -v

Add the Git upstream, which can be a URL or can be hosted on a server (in the latter case, connect with ssh).

git remote add upstream https://www.github.com/username/repo-name

Staging

When you’ve modified a file and have marked it to go in your next commit, it is considered to be a staged file.

Check the status of your Git repository, including files added that are not staged, and files that are staged.

git status

To stage modified files, use the add command, which you can run multiple times before a commit. If you make subsequent changes that you want included in the next commit, you must run add again.

You can specify the specific file with add.

git add my_script.py

With . you can add all files in the current directory including files that begin with a ..

git add .

You can remove a file from staging while retaining changes within your working directory with reset.

git reset my_script.py

Committing

Once you have staged your updates, you are ready to commit them, which will record changes you have made to the repository.

To commit staged files, you’ll run the commit command with your meaningful commit message so that you can track commits.

git commit -m "Commit message"

You can condense staging all tracked files with committing them in one step.

git commit -am "Commit message"

If you need to modify your commit message, you can do so with the --amend flag.

git commit --amend -m "New commit message"

Branches

A branch in Git is a movable pointer to one of the commits in the repository, it allows you to isolate work and manage feature development and integrations. You can learn more about branches by reading the Git documentation.

List all current branches with the branch command. An asterisk (*) will appear next to your currently active branch.

git branch

Create a new branch. You will remain on your currently active branch until you switch to the new one.

git branch new-branch

Switch to any existing branch and check it out into your current working directory.

git checkout another-branch

You can consolidate the creation and checkout of a new branch by using the -b flag.

git checkout -b new-branch

Rename your branch name.

git branch -m current-branch-name new-branch-name

Merge the specified branch’s history into the one you’re currently working in.

git merge branch-name

Abort the merge, in case there are conflicts.

git merge --abort

You can also select a particular commit to merge with cherry-pick with the string that references the specific commit.

git cherry-pick f7649d0

When you have merged a branch and no longer need the branch, you can delete it.

git branch -d branch-name

If you have not merged a branch to master, but are sure you want to delete it, you can force delete a branch.

git branch -D branch-name

Collaborate and Update

To download changes from another repository, such as the remote upstream, you’ll use fetch.

git fetch upstream

Merge the fetched commits.

git merge upstream/master

Push or transmit your local branch commits to the remote repository branch.

git push origin master

Fetch and merge any commits from the tracking remote branch.

git pull

Inspecting

Display the commit history for the currently active branch.

git log

Show the commits that changed a particular file. This follows the file regardless of file renaming.

git log --follow my_script.py

Show the commits that are on one branch and not on the other. This will show commits on a-branch that are not on b-branch.

git log a-branch..b-branch

Look at reference logs (reflog) to see when the tips of branches and other references were last updated within the repository.

git reflog

Show any object in Git via its commit string or hash in a more human-readable format.

git show de754f5

Show Changes

The git diff command shows changes between commits, branches, and more. You can read more fully about it through the Git documentation.

Compare modified files that are on the staging area.

git diff --staged

Display the diff of what is in a-branch but is not in b-branch.

git diff a-branch..b-branch

Show the diff between two specific commits.

git diff 61ce3e6..e221d9c

Stashing

Sometimes you’ll find that you made changes to some code, but before you finish you have to begin working on something else. You’re not quite ready to commit the changes you have made so far, but you don’t want to lose your work. The git stash command will allow you to save your local modifications and revert back to the working directory that is in line with the most recent HEAD commit.

Stash your current work.

git stash

See what you currently have stashed.

git stash list

Your stashes will be named stash@{0}, stash@{1}, and so on.

Show information about a particular stash.

git stash show stash@{0}

To bring the files in a current stash out of the stash while still retaining the stash, use apply.

git stash apply stash@{0}

If you want to bring files out of a stash, and no longer need the stash, use pop.

git stash pop stash@{0}

If you no longer need the files saved in a particular stash, you can drop the stash.

git stash drop stash@{0}

If you have multiple stashes saved and no longer need to use any of them, you can use clear to remove them.

git stash clear

Ignoring Files

If you want to keep files in your local Git directory, but do not want to commit them to the project, you can add these files to your .gitignore file so that they do not cause conflicts.

Use a text editor such as nano to add files to the .gitignore file.

nano .gitignore

To see examples of .gitignore files, you can look at GitHub’s .gitignore template repo.

Rebasing

A rebase allows us to move branches around by changing the commit that they are based on. With rebasing, you can squash or reword commits.

You can start a rebase by either calling the number of commits you have made that you want to rebase (5 in the case below).

git rebase -i HEAD~5

Alternatively, you can rebase based on a particular commit string or hash.

git rebase -i 074a4e5

Once you have squashed or reworded commits, you can complete the rebase of your branch on top of the latest version of the project’s upstream code.

git rebase upstream/master

To learn more about rebasing and updating, you can read How To Rebase and Update a Pull Request, which is also applicable to any type of commit.

Resetting

Sometimes, including after a rebase, you need to reset your working tree. You can reset to a particular commit, and delete all changes, with the following command.

git reset --hard 1fc6665

To force push your last known non-conflicting commit to the origin repository, you’ll need to use --force.

Warning: Force pushing to master is often frowned upon unless there is a really important reason for doing it. Use sparingly when working on your own repositories, and work to avoid this when you’re collaborating.

git push --force origin master

To remove local untracked files and subdirectories from the Git directory for a clean working branch, you can use git clean.

git clean -f -d

If you need to modify your local repository so that it looks like the current upstream master (that is, there are too many conflicts), you can perform a hard reset.

Note: Performing this command will make your local repository look exactly like the upstream. Any commits you have made but that were not pulled into the upstream will be destroyed.

git reset --hard upstream/master

Conclusion

This guide covers some of the more common Git commands you may use when managing repositories and collaborating on software.

You can learn more about open-source software and collaboration in our Introduction to Open Source tutorial series:

There are many more commands and variations that you may find useful as part of your work with Git. To learn more about all of your available options, you can run:

git --help

To receive useful information. You can also read more about Git and look at Git’s documentation from the official Git website.

Como Instalar e Configurar o GitLab no Ubuntu 16.04

2018-10-09T21:50:08Z

Introdução

O GitLab CE, ou Community Edition, é uma aplicação open source usada principalmente para hospedar repositórios Git, com recursos adicionais relacionados ao desenvolvimento, como rastreamento de problemas. Ele é projetado para ser hospedado usando a sua própria infraestrutura, e fornece flexibilidade na implantação como um repositório interno para sua equipe de desenvolvimento, publicamente como uma forma de interagir com usuários, ou até mesmo aberto como forma de os colaboradores hospedarem seus próprios projetos.

O projeto do GitLab torna relativamente simples a configuração de uma instância do GitLab em seu próprio hardware com um mecanismo de fácil instalação. Neste guia vamos cobrir como instalar e configurar o GitLab em um servidor Ubuntu 16.04.

Pré-Requisitos

Este tutorial irá assumir que você tem acesso a um novo servidor Ubuntu 16.04. Os requisitos de hardware publicados do GitLab recomendam a utilização de um servidor com:

2 núcleos
4GB de RAM

Embora você possa substituir algum espaço de swap por RAM, isso não é recomendado. Para este guia assumiremos que você tem os recursos acima, no mínimo.

Para começar, você vai precisar de um usuário não-root com acesso sudo configurado no servidor. É também uma boa ideia configurar um firewall básico para fornecer uma camada adicional de segurança. Você pode seguir os passos em nosso tutorial Configuração Inicial de servidor com Ubuntu 16.04 para obter essa configuração.

Quando tiver satisfeito os pré-requisitos acima, continue para iniciar o procedimento de instalação.

Instalando as Dependências

Antes que possamos instalar o GitLab propriamente dito, é importante instalar alguns dos softwares que ele aproveita durante a instalação e que ele usa de forma contínua. Felizmente, todos os softwares necessários podem ser facilmente instalados a partir dos repositórios padrão do Ubuntu.

Já que esta é a nossa primeira vez usando o apt durante esta sessão, podemos atualizar o índice de pacotes local e depois instalar as dependências digitando:

sudo apt-get update
sudo apt-get install ca-certificates curl openssh-server postfix

Você provavelmente já terá alguns desses softwares instalados. Para a instalação do postfix, selecione Internet Site quando solicitado. Na próxima tela, entre com o nome de domínio do seu servidor ou seu endereço IP para configurar de que forma o sistema enviará e-mail.

Instalando o GitLab

Agora que as dependências estão instaladas, podemos instalar o GitLab. Este é um processo direto que utiliza um script de instalação para configurar seu sistema com os repositórios do GitLab.

Vá para o diretório /tmp e então baixe o script de instalação:

cd /tmp
curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh

Fique à vontade para examinar o script baixado para assegurar-se de que você esteja confortável com as ações que ele irá tomar. Você pode encontrar uma versão hospedada do script aqui:.

less /tmp/script.deb.sh

Quando estiver satisfeito com a segurança do script, execute o instalador:

sudo bash /tmp/script.deb.sh

Este script irá configurar seu servidor para utilizar os repositórios mantidos pelo GitLab. Isso lhe permite gerenciar o GitLab com as mesmas ferramentas de gerenciamento de pacotes que você usa para seus outros pacotes de sistema. Quando estiver completo, você pode instalar a aplicação real do GitLab com o apt:

sudo apt-get install gitlab-ce

Isto irá instalar os componentes necessários em seu sistema.

Ajustando as Regras de Firewall

Antes de configurar o GitLab, você precisará garantir que suas regras de firewall são permissivas o suficiente para permitir o tráfego web. Se você seguiu o guia vinculado nos pré-requisitos, você terá um firewall ufw ativado.

Veja o status atual do seu firewall ativo digitando:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)

Como você pode ver, as regras atuais permitem o tráfego SSH, mas o acesso a outros serviços está restrito. Como o Gitlab é uma aplicação web, devemos permitir acesso HTTP entrante. Se você tiver um nome de domínio associado com o seu servidor GitLab, o GitLab pode também solicitar e ativar um certificado gratuito TLS/SSL a partir do projeto Let's Encrypt para proteger a instalação. Também queremos permitir o acesso HTTPS nesse caso.

um vez que o protocolo de mapeamento de portas para HTTP e HTTPS está disponível no arquivo /etc/services, podemos permitir tais tráfegos pelo nome. Se você ainda não ativou o tráfego do OpenSSH, permita também esse tráfego agora:

sudo ufw allow http
sudo ufw allow https
sudo ufw allow OpenSSH

Se você verificar com o comando ufw status novamente, você deverá ver o acesso configurado para no mínimo esses dois serviços:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
80                         ALLOW       Anywhere                  
443                        ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
80 (v6)                    ALLOW       Anywhere (v6)             
443 (v6)                   ALLOW       Anywhere (v6)

A saída acima indica que a interface web do GitLab estará acessível assim que configurarmos a aplicação.

Editando o Arquivo de Configuração do GitLab

Antes de você poder usar a aplicação, você precisa atualizar um arquivo de configuração e executar um comando de reconfiguração. Primeiro, abra o arquivo de configuração do GitLab:

sudo nano /etc/gitlab/gitlab.rb

Perto do topo está a linha de configuração external_url. Atualize-a para corresponder ao seu próprio domínio ou endereço IP. Se você tem um domínio, altere o http para https para que o GitLab redirecione automaticamente os usuários para o site protegido pelo certificado do Let´s Encrypt que estaremos solicitando.

/etc/gitlab/gitlab.rb


# If your GitLab server does not have a domain name, you will need to use an IP
# address instead of a domain and keep the protocol as `http`.
external_url 'https://seu_dominio'

Em seguida, se seu servidor GitLab tem um nome de domínio, pesquise o arquivo pela configuração letsencrypt['enable']. Descomente a linha e defina-a para true. Isso dirá ao GitLab para solicitar um certificado Let´s Encrypt para seu domínio GitLab e configurar a aplicação para servir o tráfego com ele.

Abaixo disso, procure a configuração letsencrypt['contact_emails']. Esta configuração define uma lista de endereços de e-mail que o projeto Let´s Encrypt pode utilizar para lhe contatar se houver problemas com seu domínio. É uma boa idéia descomentar e preencher isso também para que você saiba de quaisquer problemas:

/etc/gitlab/gitlab.rb


letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['sammy@seu_domínio.com']

Salve e feche o arquivo. Agora, execute o seguinte comando para reconfigurar o GitLab:

sudo gitlab-ctl reconfigure

Isso irá inicializar o GitLab utilizando as informações que ele pode encontrar sobre seu servidor. Esse é um processo completamente automatizado, portanto você não vai ter que responder a nenhuma solicitação. Se você ativou a integração com o Let´s Encrypt, um certificado deve ser configurado para o seu domínio.

Realizando a Configuração Inicial Através da Interface web

Agora que o GitLab está executando e o acesso está permitido, podemos realizar algumas configurações iniciais da aplicação através da interface web.

Visite o nome de domínio do seu servidor GitLab em seu navegador:

http://domínio_gitlab_ou_IP

Se você ativou o Let's Encrypt e usou https em seu external_url, você deve ser redirecionado para uma conexão HTTPS segura.

Em sua primeira visita, você deve ver uma solicitação inicial para definir uma senha para a conta administrativa:

Na solicitação inicial de senha, forneça e confirme uma senha segura para a conta administrativa. Clique no botão Change your password quando tiver terminado.

Você será redirecionado à página convencional de login do GitLab:

Aqui, você pode efetuar o login com a senha que você acabou de definir. As credenciais são:

Username: root
Password: [a senha que você definiu]

Entre com esses valores nos campos para os usuários existentes e clique no botão Sign in. Você será autenticado no aplicativo e levado a uma página de entrada que solicitará que você comece a adicionar projetos:

Agora você pode fazer algumas mudanças simples para configurar o GitLab da maneira que você quiser.

Ajustando suas Configurações de Perfil

Uma das primeiras coisas que você deve fazer após uma nova instalação é colocar o seu perfil na melhor forma. O GitLab seleciona alguns padrões razoáveis, mas estes geralmente não são apropriados quando você começa a usar o software.

Para fazer as modificações necessárias, clique no ícone do usuário no canto superior direito da interface. No menu suspenso exibido, selecione Settings:

Você será levado para a seção de perfil ou Profile nas suas configurações:

Ajuste Name e Email trocando "Administrator" e "admin@example.com" para algo mais adequado. O nome que você selecionou será mostrado para os outros usuários, equanto o e-mail será utilizado para detecção padrão do avatar, notificações, ações no Git através da interface, etc.

Clique no botão Update Profile setting na parte inferior quando terminar:

Um e-mail de confirmação será enviado ao endereço que você forneceu.
Siga as instruções no e-mail para confirmar sua conta para que você possa começar a utilização do GitLab.

Alterando o Nome da Sua Conta

A seguir, clique no item Account na barra de menu à esquerda:

Aqui, você pode encontrar seu token de API privada ou configurar a autenticação de dois fatores. Contudo, a funcionalidade a qual estamos interessados no momento é a seção Change username.

Por padrão, à primeira conta administrativa é dado o nome root. Como esse é um nome conhecido de conta, é mais seguro alterar esse nome para um diferente. Você ainda terá privilégios administrativos; a única coisa que irá mudar é o nome:

Clique no botão Update username para fazer a alteração:

Na próxima vez que você fizer o login no GitLab, lembre-se de utilizar seu novo nome de usuário.

Adicionando uma Chave SSH à sua Conta

Em muitos casos, você irá querer usar chaves SSH com Git para interagir com seus projetos GitLab. Para fazer isso, você precisa adicionar sua chave pública à sua conta do GitLab.

Se você já tem um par de chaves SSH criado em seu computador local, você geralmente pode ver a chave pública digitando:

cat ~/.ssh/id_rsa.pub

Você deverá ver um grande pedaço de texto, como este:

Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMuyMtMl6aWwqBCvQx7YXvZd7bCFVDsyln3yh5/8Pu23LW88VXfJgsBvhZZ9W0rPBGYyzE/TDzwwITvVQcKrwQrvQlYxTVbqZQDlmsC41HnwDfGFXg+QouZemQ2YgMeHfBzy+w26/gg480nC2PPNd0OG79+e7gFVrTL79JA/MyePBugvYqOAbl30h7M1a7EHP3IV5DQUQg4YUq49v4d3AvM0aia4EUowJs0P/j83nsZt8yiE2JEYR03kDgT/qziPK7LnVFqpFDSPC3MR3b8B354E9Af4C/JHgvglv2tsxOyvKupyZonbyr68CqSorO2rAwY/jWFEiArIaVuDiR9YM5 sammy@mydesktop

Copie esse texto e volte para a página de configurações do perfil na interface web do gitLab.

Se, em vez disso, você receber uma mensagem parecida como isto, você ainda não tem um par SSH configurado em sua máquina:

Outputcat: /home/sammy/.ssh/id_rsa.pub: No such file or directory

Se for o caso, você pode criar um par de chaves SSH digitando:

ssh-keygen

Aceite os padrões e, opcionalmente, forneça uma senha para proteger a chave localmente:

OutputGenerating public/private rsa key pair.
Enter file in which to save the key (/home/sammy/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sammy/.ssh/id_rsa.
Your public key has been saved in /home/sammy/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:I8v5/M5xOicZRZq/XRcSBNxTQV2BZszjlWaIHi5chc0 sammy@gitlab.docsthat.work
The key's randomart image is:
+---[RSA 2048]----+
|          ..%o==B|
|           *.E =.|
|        . ++= B  |
|         ooo.o . |
|      . S .o  . .|
|     . + .. .   o|
|      +   .o.o ..|
|       o .++o .  |
|        oo=+     |
+----[SHA256]-----+

Depois de ter isso, você pode exibir sua chave pública como acima, digitando:

cat ~/.ssh/id_rsa.pub

Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMuyMtMl6aWwqBCvQx7YXvZd7bCFVDsyln3yh5/8Pu23LW88VXfJgsBvhZZ9W0rPBGYyzE/TDzwwITvVQcKrwQrvQlYxTVbqZQDlmsC41HnwDfGFXg+QouZemQ2YgMeHfBzy+w26/gg480nC2PPNd0OG79+e7gFVrTL79JA/MyePBugvYqOAbl30h7M1a7EHP3IV5DQUQg4YUq49v4d3AvM0aia4EUowJs0P/j83nsZt8yiE2JEYR03kDgT/qziPK7LnVFqpFDSPC3MR3b8B354E9Af4C/JHgvglv2tsxOyvKupyZonbyr68CqSorO2rAwY/jWFEiArIaVuDiR9YM5 sammy@mydesktop

Copie o bloco de texto exibido e volte para as configurações do perfil na interface web do GitLab.

Clique no item SSH keys no menu à esquerda:

No espaço fornecido cole a chave pública que você copiou da sua máquina local. Dê a ela um título descritivo, e clique no botão Add key:

Agora você deve conseguir gerenciar seus projetos e repositórios do GitLab a partir de sua máquina local sem ter que fornecer suas credenciais de conta do GitLab.

Restringindo ou Desabilitando Inscrições Públicas (Opcional)

Você deve ter notado que é possível que alguém se inscreva para uma conta quando visitar a página de destino da sua instância do GitLab. Isso pode ser o que você deseja se estiver hospedando um projeto público. No entanto, muitas vezes, configurações mais restritivas são desejáveis.

Para começar, vá até a área administrativa clicando no ícone de chave inglesa na barra de menu principal na parte superior da página:

Na página seguinte, você pode ver uma visão geral da sua instância do GitLab como um todo. Para ajustar as configurações, clique no item Settings na parte inferior do menu à esquerda.

Você será levado para as configurações globais da sua instância do GitLab. Aqui, você pode ajustar várias configurações que afetam se novos usuários podem se inscrever e qual será o nível de acesso deles.

Desabilitando Inscrições

Se você deseja desabilitar completamente as inscrições (você ainda pode criar manualmente as contas para novos usuários), desça até a seção Sign-up Restrictions.

Desmarque a caixa de seleção Sign-up enabled:

Role para baixo até o final e clique no botão Save:

A seção de inscrição agora deve estar removida da página inicial do GitLab.

Restringindo Inscrições Por Domínio

Se você estiver usando o GitLab como parte de uma organização que fornece endereços de e-mail associados a um domínio, poderá restringir as inscrições por domínio em vez de desativá-las completamente.

Na seção Sign-up Restrictions, primeiro selecione a caixa Send confirmation email on sign-up permitindo somente que os usuários façam login depois de confirmarem seus e-mails.

Em seguida, adicione o seu domínio ou domínios à caixa Whitelisted domains for sign-ups, uma por linha. Você pode utilizar o asterisco "*" para especificar domínios curinga.

Role para baixo até o final e clique no botão Salvar:

A seção de inscrição agora deve estar removida da página inicial do GitLab.

Restringindo a Criação de Projetos

Por padrão, novos usuários podem criar até 10 projetos. Se você deseja permitir que novos usuários externos tenham visibilidade e participação, mas quer restringir seus acessos ao criar novos projetos, você pode fazer isto na seção Account and Limit Settings.

Lá dentro, você pode alterar o limite padrão de projetos em Default projects limit para 0 para desativar completamente a criação de projetos por novos usuários.

Novos usuários ainda podem ser adicionados a projetos manualmente e terão acesso a projetos internos ou públicos criados por outros usuários.

Role para baixo até o final e clique no botão Salvar:

Agora, novos usuários poderão criar contas, mas não poderão criar projetos.

Criando um Cron Job para Renovar Automaticamente os Certificados Let's Encrypt

Por padrão, os certificados Let's Encrypt são válidos por 90 dias. Se você ativou o Let's Encrypt para o seu domínio do GitLab anteriormente, você precisará garantir que seus certificados sejam renovados regularmente para evitar interrupções no serviço. O GitLab fornece o comando gitlab-ctl renew-le-certs para solicitar novos certificados quando seus ativos atuais se aproximarem da expiração.

Para automatizar este processo, podemos criar um cron job para executar automaticamente este comando regularmente. O comando somente irá renovar o certificado quando ele estiver perto da expiração, para que possamos executá-lo com segurança regularmente

Para começar, crie e abra um arquivo em /etc/cron.daily/gitlab-le em seu editor de textos:

sudo nano /etc/cron.daily/gitlab-le

Dentro dele, cole o seguinte script:

/etc/cron.daily/gitlab-le


#!/bin/bash

set -e

/usr/bin/gitlab-ctl renew-le-certs > /dev/null

Salve e feche o arquivo quando tiver terminado.

Marque o arquivo como executável digitando:

sudo chmod +x /etc/cron.daily/gitlab-le

Agora, o GitLab deve verificar automaticamente todos os dias se seu certificado Let's Encrypt precisa ser renovado. Em caso afirmativo, o comando renovará o certificado automaticamente.

Conclusão

Agora você deve ter uma instância do GitLab em funcionamento hospedada em seu próprio servidor. Você pode começar a importar ou criar novos projetos e configurar o nível apropriado de acesso para sua equipe. O GitLab está regularmente adicionando recursos e atualizando sua plataforma, por isso confira a página inicial do projeto para se manter atualizado sobre quaisquer melhorias ou avisos importantes.

Por Justin Ellingwood

How To Test Ansible Roles with Molecule on Ubuntu 18.04

2018-09-18T18:21:46Z

The author selected the Mozilla Foundation to receive a donation as part of the Write for DOnations program.

Introduction

Unit testing in Ansible is key to making sure roles function as intended. Molecule makes this process easier by allowing you to specify scenarios that test roles against different environments. Using Ansible under the hood, Molecule offloads roles to a provisioner that deploys the role in a configured environment and calls a verifier (such as Testinfra) to check for configuration drift. This ensures that your role has made all of the expected changes to the environment in that particular scenario.

In this guide, you will build an Ansible role that deploys Apache to a host and configures firewalld on CentOS 7. To test that this role works as intended, you will create a test in Molecule using Docker as a driver and Testinfra, a Python library for testing the state of servers. Molecule will provision Docker containers to test the role and Testinfra will verify that the server has been configured as intended. When you're finished, you'll be able to create multiple test cases for builds across environments and run these tests using Molecule.

Prerequisites

Before you begin this guide you'll need the following:

One Ubuntu 18.04 server. Follow the steps in the Initial Server Setup with Ubuntu 18.04 guide to create a non-root sudo user, and make sure you can connect to the server without a password.
Docker installed on your server. Follow Steps 1 and 2 in How To Install and Use Docker on Ubuntu 18.04, including adding your non-root user to the docker group.
Python 3 and venv installed and configured on your server. Follow How To Install Python 3 and Set Up a Programming Environment on an Ubuntu 18.04 Server for guidance.
Familiarity with Ansible playbooks. For review, see Configuration Management 101: Writing Ansible Playbooks.

Step 1 — Preparing the Environment

If you've followed the prerequisites, you should have Python 3, venv, and Docker installed and correctly configured. Let's begin by creating a virtual environment to test Ansible with Molecule.

Start by logging in as your non-root user and creating a new virtual environment:

python3 -m venv my_env

Activate it to ensure that your actions are restricted to that environment:

source my_env/bin/activate

Next, in your activated environment, install the wheel package, which provides the bdist_wheel setuptools extension that pip uses to install Ansible:

python3 -m pip install wheel

You can now install molecule and docker with pip. Ansible will be automatically installed as a dependency for Molecule:

python3 -m pip install molecule docker

Here is what each of these packages will do:

molecule: This is the main Molecule package that you will use to test roles. Installing molecule automatically installs Ansible, along with other dependencies, and enables the use of Ansible playbooks to execute roles and tests.
docker: This Python library is used by Molecule to interface with Docker. You will need this since you're using Docker as a driver.

Next, let's create a role in Molecule.

Step 2 — Creating a Role in Molecule

With your environment set up, you can use Molecule to create a basic role that you will use to test an installation of Apache. This role will create the directory structure and some initial tests, and specify Docker as the driver so that Molecule uses Docker to run its tests.

Create a new role called ansible-apache:

molecule init role -r ansible-apache -d docker

The -r flag specifies the name of the role while -d specifies the driver, which provisions the hosts for Molecule to use in testing.

Change into the directory of the newly created role:

cd ansible-apache

Test the default role to check if Molecule has been set up properly:

molecule test

You will see output that lists each of the default test actions. Before starting the test, Molecule validates the configuration file molecule.yml to make sure everything is in order. It also prints this test matrix, which specifies the order of test actions:

Output--> Validating schema /home/sammy/ansible-apache/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy
...

We will discuss each test action in detail once you've created your role and customized your tests. For now, pay attention to the PLAY_RECAP for each test, and be sure that none of the default actions returns a failed status. For example, the PLAY_RECAP for the default 'create' action should look like this:

Output...
PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=4    unreachable=0    failed=0

Let's move on to modifying the role to configure Apache and firewalld.

Step 3 — Configuring Apache and Firewalld

To configure Apache and firewalld, you will create a tasks file for the role, specifying packages to install and services to enable. These details will be extracted from a variables file and template that you will use to replace the default Apache index page.

Still in the ansible-apache directory, create a tasks file for the role using nano or your favorite text editor:

nano tasks/main.yml

You'll see that the file already exists. Delete what's there and replace it with the following code to install the required packages and enable the correct services, HTML defaults, and firewall settings:

~/ansible-apache/tasks/main.yml

---
- name: "Ensure required packages are present"
  yum:
    name: "{{ pkg_list }}"
    state: present

- name: "Ensure latest index.html is present"
  template:
    src: index.html.j2
    dest: /var/www/html/index.html

- name: "Ensure httpd service is started and enabled"
  service:
    name: "{{ item }}"
    state: started
    enabled: true
  with_items: "{{ svc_list }}"

- name: "Whitelist http in firewalld"
  firewalld:
    service: http
    state: enabled
    permanent: true
    immediate: true

This playbook includes 4 tasks:

"Ensure required packages are present": This task will install the packages listed in the variables file under pkg_list. The variables file will be located at ~/ansible-apache/vars/main.yml and you will create it at the end of this step.
"Ensure latest index.html is present": This task will copy a template page, index.html.j2, and paste it over the default index file, /var/www/html/index.html, generated by Apache. You will also create the new template in this step.
"Ensure httpd service is started and enabled": This task will start and enable the services listed in svc_list in the variables file.
"Whitelist http in firewalld": This task will whitelist the http service in firewalld. Firewalld is a complete firewall solution present by default on CentOS servers. For the http service to work, you will need to expose the required ports. Instructing firewalld to whitelist a service ensures that it whitelists all of the ports that the service requires.

Save and close the file when you are finished.

Next, let's create a templates directory for the index.html.j2 template page:

mkdir templates

Create the page itself:

nano templates/index.html.j2

Paste in the following boilerplate code:

~/ansible-apache/templates/index.html.j2

<div style="text-align: center">
    <h2>Managed by Ansible</h2>
</div>

Save and close the file.

The final step in completing the role is writing the variables file, which provides the names of packages and services to our main role playbook:

nano vars/main.yml

Paste over the default content with the following code, which specifies pkg_list and svc_list:

~/ansible-apache/vars/main.yml

---
pkg_list:
  - httpd
  - firewalld
svc_list:
  - httpd
  - firewalld

These lists contain the following information:

pkg_list: This contains the names of the packages that the role will install: httpd and firewalld.
svc_list: This contains the names of the services that the role will start and enable: httpd and firewalld.

Note: Make sure that your variables file doesn't have any blank lines or your test will fail during linting.

Now that you've finished creating your role, let's configure Molecule to test if it works as intended.

Step 4 — Modifying the Role for Running Tests

In our case, configuring Molecule involves modifying the Molecule configuration file molecule.yml to add platform specifications. Because you're testing a role that configures and starts the httpd systemd service, you will need to use an image with systemd configured and privileged mode enabled. For this tutorial, you will use the milcom/centos7-systemd image available on Docker Hub. Privileged mode allows containers to run with almost all of the capabilities of their host machine.

Let's edit molecule.yml to reflect these changes:

nano molecule/default/molecule.yml

Add the highlighted platform information:

~/ansible-apache/molecule/default/molecule.yml

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: centos7
    image: milcom/centos7-systemd
    privileged: true
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

Save and close the file when you are done.

Now that you've successfully configured the test environment, let's move on to writing the test cases that Molecule will run against your container after executing the role.

Step 5 — Writing Test Cases

In the test for this role, you will check the following conditions:

That the httpd and firewalld packages are installed.
That the httpd and firewalld services are running and enabled.
That the http service is enabled in your firewall settings.
That index.html contains the same data specified in your template file.

If all of these tests pass, then the role works as intended.

To write the test cases for these conditions, let's edit the default tests in ~/ansible-apache/molecule/default/tests/test_default.py. Using Testinfra, we will write the test cases as Python functions that use Molecule classes.

Open test_default.py:

nano molecule/default/tests/test_default.py

Delete the contents of the file so that you can write the tests from scratch.

Note: As you write your tests, make sure that they are separated by two new lines or they will fail.

Start by importing the required Python modules:

~/ansible-apache/molecule/default/tests/test_default.py

import os
import pytest

import testinfra.utils.ansible_runner

These modules include:

os: This built-in Python module enables operating-system-dependent functionality, making it possible for Python to interface with the underlying operating system.
pytest: The pytest module enables test writing.
testinfra.utils.ansible_runner: This Testinfra module uses Ansible as the backend for command execution.

Under the module imports, add the following code, which uses the Ansible backend to return the current host instance:

~/ansible-apache/molecule/default/tests/test_default.py

...
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')

With your test file configured to use the Ansible backend, let's write unit tests to test the state of the host.

The first test will ensure that httpd and firewalld are installed:

~/ansible-apache/molecule/default/tests/test_default.py

...

@pytest.mark.parametrize('pkg', [
  'httpd',
  'firewalld'
])
def test_pkg(host, pkg):
    package = host.package(pkg)

    assert package.is_installed

The test begins with the pytest.mark.parametrize decorator, which allows us to parameterize the arguments for the test. This first test will take test_pkg as a parameter to test for the presence of the httpd and firewalld packages.

The next test checks whether or not httpd and firewalld are running and enabled. It takes test_svc as a parameter:

~/ansible-apache/molecule/default/tests/test_default.py

...

@pytest.mark.parametrize('svc', [
  'httpd',
  'firewalld'
])
def test_svc(host, svc):
    service = host.service(svc)

    assert service.is_running
    assert service.is_enabled

The last test checks that the files and contents passed to parametrize() exist. If the file isn't created by your role and the content isn't set properly, assert will return False:

~/ansible-apache/molecule/default/tests/test_default.py

...

@pytest.mark.parametrize('file, content', [
  ("/etc/firewalld/zones/public.xml", "<service name=\"http\"/>"),
  ("/var/www/html/index.html", "Managed by Ansible")
])
def test_files(host, file, content):
    file = host.file(file)

    assert file.exists
    assert file.contains(content)

In each test, assert will return True or False depending on the test result.

The finished file looks like this:

~/ansible-apache/molecule/default/tests/test_default.py

import os
import pytest

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


@pytest.mark.parametrize('pkg', [
  'httpd',
  'firewalld'
])
def test_pkg(host, pkg):
    package = host.package(pkg)

    assert package.is_installed


@pytest.mark.parametrize('svc', [
  'httpd',
  'firewalld'
])
def test_svc(host, svc):
    service = host.service(svc)

    assert service.is_running
    assert service.is_enabled


@pytest.mark.parametrize('file, content', [
  ("/etc/firewalld/zones/public.xml", "<service name=\"http\"/>"),
  ("/var/www/html/index.html", "Managed by Ansible")
])
def test_files(host, file, content):
    file = host.file(file)

    assert file.exists
    assert file.contains(content)

Now that you've specified your test cases, let's test the role.

Step 6 — Testing the Role with Molecule

Once you initiate the test, Molecule will execute the actions you defined in your scenario. Let's now run the default molecule scenario again, executing the actions in the default test sequence while looking more closely at each.

Run the test for the default scenario again:

molecule test

This will initiate the test run. The initial output prints the default test matrix:

Output--> Validating schema /home/sammy/ansible-apache/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

Let's go through each test action and the expected output, starting with linting.

The linting action executes yamllint, flake8, and ansible-lint:

yamllint: This linter is executed on all YAML files present in the role directory.
flake8: This Python code linter checks tests created for Testinfra.
ansible-lint: This linter for Ansible playbooks is executed in all scenarios.

Output...
--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/sammy/ansible-apache/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/sammy/ansible-apache/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/sammy/ansible-apache/molecule/default/playbook.yml...
Lint completed successfully.

The next action, destroy, is executed using the destroy.yml file. This is done to test our role on a newly created container.

By default, destroy is called twice: at the start of the test run, to delete any pre-existing containers, and at the end, to delete the newly created container:

Output...
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) deletion to complete] *******************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Delete docker network(s)] ************************************************
    skipping: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=1    unreachable=0    failed=0

After the destroy action is complete, the test will move on to dependency. This action allows you to pull dependencies from ansible-galaxy if your role requires them. In this case, our role does not:

Output...
--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.

The next test action is a syntax check, which is executed on the default playbook.yml playbook. It works in a similar way to the --syntax-check flag in the command ansible-playbook --syntax-check playbook.yml:

Output...
--> Scenario: 'default'
--> Action: 'syntax'

    playbook: /home/sammy/ansible-apache/molecule/default/playbook.yml

Next, the test moves on to the create action. This uses the create.yml file in your role's Molecule directory to create a Docker container with your specifications:

Output...

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Log into a Docker registry] **********************************************
    skipping: [localhost] => (item=None)
    skipping: [localhost]

    TASK [Create Dockerfiles from image names] *************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Discover local Docker images] ********************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Build an Ansible compatible image] ***************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Create docker network(s)] ************************************************
    skipping: [localhost]

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) creation to complete] *******************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=4    unreachable=0    failed=0

After create, the test moves on to the prepare action. This action executes the prepare playbook, which brings the host to a specific state before running converge. This is useful if your role requires a pre-configuration of the system before the role is executed. Again, this does not apply to our role:

Output...
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, prepare playbook not configured.

After prepare, the converge action executes your role on the container by running the playbook.yml playbook. If multiple platforms are configured in the molecule.yml file, Molecule will converge on all of these:

Output...
--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [centos7]

    TASK [ansible-apache : Ensure required packages are present] *******************
    changed: [centos7]

    TASK [ansible-apache : Ensure latest index.html is present] ********************
    changed: [centos7]

    TASK [ansible-apache : Ensure httpd service is started and enabled] ************
    changed: [centos7] => (item=httpd)
    changed: [centos7] => (item=firewalld)

    TASK [ansible-apache : Whitelist http in firewalld] ****************************
    changed: [centos7]

    PLAY RECAP *********************************************************************
    centos7                    : ok=5    changed=4    unreachable=0    failed=0

After coverge, the test moves on to idempotence. This action tests the playbook for idempotence to make sure no unexpected changes are made in multiple runs:

Output...
--> Scenario: 'default'
--> Action: 'idempotence'
Idempotence completed successfully.

The next test action is the side-effect action. This lets you produce situations in which you'll be able to test more things, like HA failover. By default, Molecule doesn't configure a side-effect playbook and the task is skipped:

Output...
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side effect playbook not configured.

Molecule will then run the verifier action using the default verifier, Testinfra. This action executes the tests you wrote earlier in test_default.py. If all the tests pass successfully, you will see a success message and Molecule will proceed to the next step:

Output...
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/sammy/ansible-apache/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux -- Python 3.6.5, pytest-3.7.3, py-1.5.4, pluggy-0.7.1
    rootdir: /home/sammy/ansible-apache/molecule/default, inifile:
    plugins: testinfra-1.14.1
collected 6 items

    tests/test_default.py ......                                             [100%]

    ========================== 6 passed in 41.05 seconds ===========================
Verifier completed successfully.

Finally, Molecule destroys the instances completed during the test and deletes the network assigned to those instances:

Output...
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) deletion to complete] *******************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Delete docker network(s)] ************************************************
    skipping: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=2    unreachable=0    failed=0

The test actions are now complete, verifying that your role worked as intended.

Conclusion

In this article you created an Ansible role to install and configure Apache and firewalld. You then wrote unit tests with Testinfra that Molecule used to assert that the role ran successfully.

You can use the same basic method for highly complex roles, and automate testing using a CI pipeline as well. Molecule is a highly configurable tool that can be used to test roles with any providers that Ansible supports, not just Docker. It's also possible to automate testing against your own infrastructure, making sure that your roles are always up-to-date and functional. The official Molecule documentation is the best resource for learning how to use Molecule.

How To Provision and Manage Remote Docker Hosts with Docker Machine on Ubuntu 18.04

2018-10-01T20:06:38Z

Introduction

Docker Machine is a tool that makes it easy to provision and manage multiple Docker hosts remotely from your personal computer. Such servers are commonly referred to as Dockerized hosts and are used to run Docker containers.

While Docker Machine can be installed on a local or a remote system, the most common approach is to install it on your local computer (native installation or virtual machine) and use it to provision Dockerized remote servers.

Though Docker Machine can be installed on most Linux distributions as well as macOS and Windows, in this tutorial, you'll install it on your local machine running Ubuntu 18.04 and use it to provision Dockerized DigitalOcean Droplets. If you don't have a local Ubuntu 18.04 machine, you can follow these instructions on any Ubuntu 18.04 server.

Prerequisites

To follow this tutorial, you will need the following:

A local machine or server running Ubuntu 18.04 with Docker installed. See How To Install and Use Docker on Ubuntu 18.04 for instructions.
A DigitalOcean API token. If you don't have one, generate it using this guide. When you generate a token, be sure that it has read-write scope. That is the default, so if you do not change any options while generating it, it will have read-write capabilities.

Step 1 — Installing Docker Machine

In order to use Docker Machine, you must first install it locally. On Ubuntu, this means downloading a handful of scripts from the official Docker repository on GitHub.

To download and install the Docker Machine binary, type:

wget https://github.com/docker/machine/releases/download/v0.15.0/docker-machine-$(uname -s)-$(uname -m)

The name of the file should be docker-machine-Linux-x86_64. Rename it to docker-machine to make it easier to work with:

mv docker-machine-Linux-x86_64 docker-machine

Make it executable:

chmod +x docker-machine

Move or copy it to the /usr/local/bin directory so that it will be available as a system command:

sudo mv docker-machine /usr/local/bin

Check the version, which will indicate that it's properly installed:

docker-machine version

You'll see output similar to this, displaying the version number and build:

Output
docker-machine version 0.15.0, build b48dc28d

Docker Machine is installed. Let's install some additional helper tools to make Docker Machine easier to work with.

Step 2 — Installing Additional Docker Machine Scripts

There are three Bash scripts in the Docker Machine GitHub repository you can install to make working with the docker and docker-machine commands easier. When installed, these scripts provide command completion and prompt customization.

In this step, you'll install these three scripts into the /etc/bash_completion.d directory on your local machine by downloading them directly from the Docker Machine GitHub repository.

Note: Before downloading and installing a script from the internet in a system-wide location, you should inspect the script's contents first by viewing the source URL in your browser.

The first script allows you to see the active machine in your prompt. This comes in handy when you are working with and switching between multiple Dockerized machines. The script is called docker-machine-prompt.bash. Download it

sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine-prompt.bash -O /etc/bash_completion.d/docker-machine-prompt.bash

To complete the installation of this file, you'll have to modify the value for the PS1 variable in your .bashrc file. The PS1 variable is a special shell variable used to modify the Bash command prompt. Open ~/.bashrc in your editor:

nano ~/.bashrc

Within that file, there are three lines that begin with PS1. They should look just like these:

~/.bashrc

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

...

PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

...

PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"

For each line, insert $(__docker_machine_ps1 " [%s]") near the end, as shown in the following example:

~/.bashrc

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]$(__docker_machine_ps1 " [%s]")\$ '

...

PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w$(__docker_machine_ps1 " [%s]")\$ '

...

PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$(__docker_machine_ps1 " [%s]")$PS1"

Save and close the file.

The second script is called docker-machine-wrapper.bash. It adds a use subcommand to the docker-machine command, making it significantly easier to switch between Docker hosts. To download it, type:

sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine-wrapper.bash -O /etc/bash_completion.d/docker-machine-wrapper.bash

The third script is called docker-machine.bash. It adds bash completion for docker-machine commands. Download it using:

sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine.bash -O /etc/bash_completion.d/docker-machine.bash

To apply the changes you've made so far, close, then reopen your terminal. If you're logged into the machine via SSH, exit the session and log in again, and you'll have command completion for the docker and docker-machine commands.

Let's test things out by creating a new Docker host with Docker Machine.

Step 3 — Provisioning a Dockerized Host Using Docker Machine

Now that you have Docker and Docker Machine running on your local machine, you can provision a Dockerized Droplet on your DigitalOcean account using Docker Machine's docker-machine create command. If you've not done so already, assign your DigitalOcean API token to an environment variable:

export DOTOKEN=your-api-token

NOTE: This tutorial uses DOTOKEN as the bash variable for the DO API token. The variable name does not have to be DOTOKEN, and it does not have to be in all caps.

To make the variable permanent, put it in your ~/.bashrc file. This step is optional, but it is necessary if you want to the value to persist across shell sessions.

Open that file with nano:

nano ~/.bashrc

Add this line to the file:

~/.bashrc

export DOTOKEN=your-api-token

To activate the variable in the current terminal session, type:

source ~/.bashrc

To call the docker-machine create command successfully you must specify the driver you wish to use, as well as a machine name. The driver is the adapter for the infrastructure you're going to create. There are drivers for cloud infrastructure providers, as well as drivers for various virtualization platforms.

We'll use the digitalocean driver. Depending on the driver you select, you'll need to provide additional options to create a machine. The digitalocean driver requires the API token (or the variable that evaluates to it) as its argument, along with the name for the machine you want to create.

To create your first machine, type this command to create a DigitalOcean Droplet called docker-01:

docker-machine create --driver digitalocean --digitalocean-access-token $DOTOKEN docker-01

You'll see this output as Docker Machine creates the Droplet:

Output ...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env ubuntu1804-docker

Docker Machine creates an SSH key pair for the new host so it can access the server remotely. The Droplet is provisioned with an operating system and Docker is installed. When the command is complete, your Docker Droplet is up and running.

To see the newly-created machine from the command line, type:

docker-machine ls

The output will be similar to this, indicating that the new Docker host is running:

OutputNAME        ACTIVE   DRIVER         STATE     URL                         SWARM   DOCKER        ERRORS
docker-01   -        digitalocean   Running   tcp://209.97.155.178:2376           v18.06.1-ce

Now let's look at how to specify the operating system when we create a machine.

Step 4 — Specifying the Base OS and Droplet Options When Creating a Dockerized Host

By default, the base operating system used when creating a Dockerized host with Docker Machine is supposed to be the latest Ubuntu LTS. However, at the time of this publication, the docker-machine create command is still using Ubuntu 16.04 LTS as the base operating system, even though Ubuntu 18.04 is the latest LTS edition. So if you need to run Ubuntu 18.04 on a recently-provisioned machine, you'll have to specify Ubuntu along with the desired version by passing the --digitalocean-image flag to the docker-machine create command.

For example, to create a machine using Ubuntu 18.04, type:

docker-machine create --driver digitalocean --digitalocean-image ubuntu-18-04-x64 --digitalocean-access-token $DOTOKEN docker-ubuntu-1804

You're not limited to a version of Ubuntu. You can create a machine using any operating system supported on DigitalOcean. For example, to create a machine using Debian 8, type:

docker-machine create --driver digitalocean --digitalocean-image debian-8-x64 --digitalocean-access-token $DOTOKEN docker-debian

To provision a Dockerized host using CentOS 7 as the base OS, specify centos-7-0-x86 as the image name, like so:

docker-machine create --driver digitalocean --digitalocean-image centos-7-0-x64 --digitalocean-access-token $DOTOKEN docker-centos7

The base operating system is not the only choice you have. You can also specify the size of the Droplet. By default, it is the smallest Droplet, which has 1 GB of RAM, a single CPU, and a 25 GB SSD.

Find the size of the Droplet you want to use by looking up the corresponding slug in the DigitalOcean API documentation.

For example, to provision a machine with 2 GB of RAM, two CPUs, and a 60 GB SSD, use the slug s-2vcpu-2gb:

docker-machine create --driver digitalocean --digitalocean-size s-2vcpu-2gb --digitalocean-access-token $DOTOKEN docker-03

To see all the flags specific to creating a Docker Machine using the DigitalOcean driver, type:

docker-machine create --driver digitalocean -h

Tip: If you refresh the Droplet page of your DigitalOcean dashboard, you will see the new machines you created using the docker-machine command.

Now let's explore some of the other Docker Machine commands.

Step 5 — Executing Additional Docker Machine Commands

You've seen how to provision a Dockerized host using the create subcommand, and how to list the hosts available to Docker Machine using the ls subcommand. In this step, you'll learn a few more useful subcommands.

To obtain detailed information about a Dockerized host, use the inspect subcommand, like so:

docker-machine inspect docker-01

The output includes lines like the ones in the following output. The Image line reveals the version of the Linux distribution used and the Size line indicates the size slug:

Output...
{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "203.0.113.71",
        "MachineName": "docker-01",
        "SSHUser": "root",
        "SSHPort": 22,
        ...
        "Image": "ubuntu-16-04-x64",
        "Size": "s-1vcpu-1gb",
        ...
    },

---

To print the connection configuration for a host, type:

docker-machine config docker-01

The output will be similar to this:

Output--tlsverify
--tlscacert="/home/kamit/.docker/machine/certs/ca.pem"
--tlscert="/home/kamit/.docker/machine/certs/cert.pem"
--tlskey="/home/kamit/.docker/machine/certs/key.pem"
-H=tcp://203.0.113.71:2376

The last line in the output of the docker-machine config command reveals the IP address of the host, but you can also get that piece of information by typing:

docker-machine ip docker-01

If you need to power down a remote host, you can use docker-machine to stop it:

docker-machine stop docker-01

Verify that it is stopped:

docker-machine ls

The output shows that the status of the machine has changed:

OuputNAME        ACTIVE   DRIVER         STATE     URL   SWARM   DOCKER    ERRORS
docker-01   -        digitalocean   Stopped                 Unknown

To start it again, use the start subcommand:

docker-machine start docker-01

Then review its status again:

docker-machine ls

You will see that the STATE is now set Running for the host:

OuputNAME                ACTIVE   DRIVER         STATE     URL                        SWARM   DOCKER    ERRORS

docker-01           -        digitalocean   Running   tcp://203.0.113.71:2376            v18.06.1-ce

Next let's look at how to interact with the remote host using SSH.

Step 6 — Executing Commands on a Dockerized Host via SSH

At this point, you've been getting information about your machines, but you can do more than that. For example, you can execute native Linux commands on a Docker host by using the ssh subcommand of docker-machine from your local system. This section explains how to perform ssh commands via docker-machine as well as how to open an SSH session to a Dockerized host.

Assuming that you've provisioned a machine with Ubuntu as the operating system, execute the following command from your local system to update the package database on the Docker host:

docker-machine ssh docker-01 apt-get update

You can even apply available updates using:

docker-machine ssh docker-01 apt-get upgrade

Not sure what kernel your remote Docker host is using? Type the following:

docker-machine ssh docker-01 uname -r

Finally, you can log in to the remote host with the docker machine ssh command:

docker-machine ssh docker-01

You'll be logged in as the root user and you'll see something similar to the following:

Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-131-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

14 packages can be updated.
10 updates are security updates.

Log out by typing exit to return to your local machine.

Next, we'll direct Docker's commands at our remote host.

Step 7 — Activating a Dockerized Host

Activating a Docker host connects your local Docker client to that system, which makes it possible to run normal docker commands on the remote system.

First, use Docker Machine to create a new Docker host called docker-ubuntu using Ubuntu 18.04:

docker-machine create --driver digitalocean --digitalocean-image ubuntu-18-04-x64 --digitalocean-access-token $DOTOKEN docker-ubuntu

To activate a Docker host, type the following command:

eval $(docker-machine env machine-name)

Alternatively, you can activate it by using this command:

docker-machine use machine-name

Tip When working with multiple Docker hosts, the docker-machine use command is the easiest method of switching from one to the other.

After typing any of these commands, your prompt will change to indicate that your Docker client is pointing to the remote Docker host. It will take this form. The name of the host will be at the end of the prompt:

username@localmachine:~ [docker-01]$

Now any docker command you type at this command prompt will be executed on that remote host.

Execute docker-machine ls again:

docker-machine ls

You'll see an asterisk under the ACTIVE column for docker-01:

OutputNAME                ACTIVE   DRIVER         STATE     URL                         SWARM   DOCKER    ERRORS
docker-01           *        digitalocean   Running   tcp://203.0.113.71:2376             v18.06.1-ce

To exit from the remote Docker host, type the following:

docker-machine use -u

Your prompt will no longer show the active host.

Now let's create containers on the remote machine.

Step 8 — Creating Docker Containers on a Remote Dockerized Host

So far, you have provisioned a Dockerized Droplet on your DigitalOcean account and you've activated it — that is, your Docker client is pointing to it. The next logical step is to spin up containers on it. As an example, let's try running the official Nginx container.

Use docker-machine use to select your remote machine:

docker-machine use docker-01

Now execute this command to run an Nginx container on that machine:

docker run -d -p 8080:80 --name httpserver nginx

In this command, we're mapping port 80 in the Nginx container to port 8080 on the Dockerized host so that we can access the default Nginx page from anywhere.

Once the container builds, you will be able to access the default Nginx page by pointing your web browser to http://docker_machine_ip:8080.

While the Docker host is still activated (as seen by its name in the prompt), you can list the images on that host:

docker images

The output includes the Nginx image you just used:

OutputREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               latest              71c43202b8ac        3 hours ago         109MB

You can also list the active or running containers on the host:

docker ps

If the Nginx container you ran in this step is the only active container, the output will look like this:

OutputCONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS
 PORTS                  NAMES
d3064c237372        nginx               "nginx -g 'daemon of…"   About a minute ago   Up About a minute
 0.0.0.0:8080->80/tcp   httpserver

If you intend to create containers on a remote machine, your Docker client must be pointing to it — that is, it must be the active machine in the terminal that you're using. Otherwise you'll be creating the container on your local machine. Again, let your command prompt be your guide.

Docker Machine can create and manage remote hosts, and it can also remove them.

Step 9 – Removing Docker Hosts

You can use Docker Machine to remove a Docker host you've created. Use the docker-machine rm command to remove the docker-01 host you created:

docker-machine rm docker-01

The Droplet is deleted along with the SSH key created for it. List the hosts again:

docker-machine ls

This time, you won't see the docker-01 host listed in the output. And if you've only created one host, you won't see any output at all.

Be sure to execute the command docker-machine use -u to point your local Docker daemon back to your local machine.

Step 10 — Disabling Crash Reporting (Optional)

By default, whenever an attempt to provision a Dockerized host using Docker Machine fails, or Docker Machine crashes, some diagnostic information is sent to a Docker account on Bugsnag. If you're not comfortable with this, you can disable the reporting by creating an empty file called no-error-report in your local computer's .docker/machine directory.

To create the file, type:

touch ~/.docker/machine/no-error-report

Check the file for error messages if provisioning fails or Docker Machine crashes.

Conclusion

You've installed Docker Machine and used it to provision multiple Docker hosts on DigitalOcean remotely from your local system. From here you should be able to provision as many Dockerized hosts on your DigitalOcean account as you need.

For more on Docker Machine, visit the official documentation page. The three Bash scripts downloaded in this tutorial are hosted on this GitHub page.

How To Back Up and Restore a Kubernetes Cluster on DigitalOcean Using Heptio Ark

2018-09-28T21:21:05Z

Introduction

Heptio Ark is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster's Persistent Volumes using your cloud provider's block storage snapshot features, and can then restore your cluster's objects and Persistent Volumes to a previous state.

StackPointCloud's DigitalOcean Ark Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster's state and restore it should disaster strike.

In this tutorial we'll set up and configure the Ark client on a local machine, and deploy the Ark server into our Kubernetes cluster. We'll then deploy a sample Nginx app that uses a Persistent Volume for logging, and simulate a disaster recovery scenario.

Prerequisites

Before you begin this tutorial, you should have the following available to you:

On your local computer:

The kubectl command-line tool, configured to connect to your cluster. You can read more about installing and configuring kubectl in the official Kubernetes documentation.
The git command-line utility. You can learn how to install git in Getting Started with Git.

In your DigitalOcean account:

A DigitalOcean Kubernetes cluster, or a Kubernetes cluster (version 1.7.5 or later) on DigitalOcean Droplets
A DNS server running inside of your cluster. If you are using DigitalOcean Kubernetes, this is running by default. To learn more about configuring a Kubernetes DNS service, consult Customizing DNS Service from the official Kuberentes documentation.
A DigitalOcean Space that will store your backed-up Kubernetes objects. To learn how to create a Space, consult the Spaces product documentation.
An access key pair for your DigitalOcean Space. To learn how to create a set of access keys, consult How to Manage Administrative Access to Spaces.
A personal access token for use with the DigitalOcean API. To learn how to create a personal access token, consult How to Create a Personal Access Token.

Once you have all of this set up, you're ready to begin with this guide.

Step 1 — Installing the Ark Client

The Heptio Ark backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we'll install the local Ark client.

In your web browser, navigate to the Ark GitHub repo releases page, find the latest release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we'll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine.

Then, from the command line on your local computer, navigate to the temporary /tmp directory and cd into it:

cd /tmp

Use wget and the link you copied earlier to download the release tarball:

wget https://link_copied_from_release_page

Once the download completes, extract the tarball using tar (note the filename may differ depending on the current release version and your OS):

tar -xvzf ark-v0.9.6-linux-amd64.tar.gz

The /tmp directory should now contain the extracted ark binary as well as the tarball you just downloaded.

Verify that you can run the ark client by executing the binary:

./ark --help

You should see the following help output:

OutputHeptio Ark is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

If you're familiar with kubectl, Ark supports a similar model, allowing you to
execute commands such as 'ark get backup' and 'ark create schedule'. The same
operations can also be performed as 'ark backup get' and 'ark schedule create'.

Usage:
  ark [command]

Available Commands:
  backup      Work with backups
  client      Ark client related commands
  completion  Output shell completion code for the specified shell (bash or zsh)
  create      Create ark resources
  delete      Delete ark resources
  describe    Describe ark resources
  get         Get ark resources
  help        Help about any command
  plugin      Work with plugins
  restic      Work with restic
  restore     Work with restores
  schedule    Work with schedules
  server      Run the ark server
  version     Print the ark version and associated image

. . .

At this point you should move the ark executable out of the temporary /tmp directory and add it to your PATH. To add it to your PATH on an Ubuntu system, simply copy it to /usr/local/bin:

sudo mv ark /usr/local/bin/ark

You're now ready to configure the Ark server and deploy it to your Kubernetes cluster.

Step 2 — Installing and Configuring the Ark Server

Before we deploy Ark into our Kubernetes cluster, we'll first create Ark's prerequisite objects. Ark's prerequisites consist of:

A heptio-ark Namespace
The ark Service Account
Role-based access control (RBAC) rules to grant permissions to the ark Service Account
Custom Resources (CRDs) for the Ark-specific resources: Backup, Schedule, Restore, Config

A YAML file containing the specs for the above Kubernetes objects can be found in the official Ark Git repository. While still in the /tmp directory, download the Ark repo using git:

git clone https://github.com/heptio/ark.git

Once downloaded, navigate into the ark directory:

cd ark

The prerequisite resources listed above can be found in the examples/common/00-prereqs.yaml YAML file. We'll create these resources in our Kubernetes cluster by using kubectl apply and passing in the file:

kubectl apply -f examples/common/00-prereqs.yaml

You should see the following output:

Outputcustomresourcedefinition.apiextensions.k8s.io/backups.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/schedules.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/restores.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/configs.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/downloadrequests.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/deletebackuprequests.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/podvolumebackups.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/podvolumerestores.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/resticrepositories.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/backupstoragelocations.ark.heptio.com created
namespace/heptio-ark created
serviceaccount/ark created
clusterrolebinding.rbac.authorization.k8s.io/ark created

Now that we've created the necessary Ark Kubernetes objects in our cluster, we can download and install the Ark DigitalOcean Plugin, which will allow us to use DigitalOcean Spaces as a backupStorageProvider (for Kubernetes objects), and DigitalOcean Block Storage as a persistentVolumeProvider (for Persistent Volume backups).

Move back out of the ark directory and fetch the plugin from StackPointCloud's repo using git:

cd ..
git clone https://github.com/StackPointCloud/ark-plugin-digitalocean.git

Move into the plugin directory:

cd ark-plugin-digitalocean

We'll now save the access keys for our DigitalOcean Space as a Kubernetes Secret. First, open up the examples/credentials-ark file using your favorite editor:

nano examples/credentials-ark

Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your Spaces access key and secret key:

examples/credentials-ark

[default]
aws_access_key_id=your_spaces_access_key_here
aws_secret_access_key=your_spaces_secret_key_here

Save and close the file.

Now, create the cloud-credentials Secret using kubectl, inserting your Personal Access Token using the digitalocean_token data item:

kubectl create secret generic cloud-credentials \
    --namespace heptio-ark \
    --from-file cloud=examples/credentials-ark \
    --from-literal digitalocean_token=your_personal_access_token

You should see the following output:

Outputsecret/cloud-credentials created

To confirm that the cloud-credentials Secret was created successfully, you can describe it using kubectl:

kubectl describe secrets/cloud-credentials --namespace heptio-ark

You should see the following output describing the cloud-credentials secret:

OutputName:         cloud-credentials
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cloud:               115 bytes
digitalocean_token:  64 bytes

We can now move on to creating an Ark Config object named default. To do this, we'll edit a YAML configuration file and then create the object in our Kubernetes cluster.

Open examples/10-ark-config.yaml in your favorite editor:

nano examples/10-ark-config.yaml

Insert your Space's name and region in the highlighted fields:

examples/10-ark-config.yaml

---
apiVersion: ark.heptio.com/v1
kind: Config
metadata:
  namespace: heptio-ark
  name: default
persistentVolumeProvider:
  name: digitalocean
backupStorageProvider:
  name: aws
  bucket: space_name_here
  config:
    region: space_region_here
    s3ForcePathStyle: "true"
    s3Url: https://space_region_here.digitaloceanspaces.com
backupSyncPeriod: 30m
gcSyncPeriod: 30m
scheduleSyncPeriod: 1m
restoreOnlyMode: false

persistentVolumeProvider sets DigitalOcean Block Storage as the the provider for Persistent Volume backups. These will be Block Storage Volume Snapshots.

backupStorageProvider sets DigitalOcean Spaces as the provider for Kubernetes object backups. Ark will create a tarball of all your Kubernetes objects (or some, depending on how you execute it), and upload this tarball to Spaces.

When you're done, save and close the file.

Create the object in your cluster using kubectl apply:

kubectl apply -f examples/10-ark-config.yaml

You should see the following output:

Outputconfig.ark.heptio.com/default created

At this point, we've finished configuring the Ark server and can create its Kubernetes deployment, found in the examples/20-deployment.yaml configuration file. Let's take a quick look at this file:

cat examples/20-deployment.yaml

You should see the following text:

examples/20-deployment.yaml

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: heptio-ark
  name: ark
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: ark
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8085"
        prometheus.io/path: "/metrics"
    spec:
      restartPolicy: Always
      serviceAccountName: ark
      containers:
        - name: ark
          image: gcr.io/heptio-images/ark:latest
          command:
            - /ark
          args:
            - server
          volumeMounts:
            - name: cloud-credentials
              mountPath: /credentials
            - name: plugins
              mountPath: /plugins
            - name: scratch
              mountPath: /scratch
          env:
            - name: AWS_SHARED_CREDENTIALS_FILE
              value: /credentials/cloud
            - name: ARK_SCRATCH_DIR
              value: /scratch
            - name: DIGITALOCEAN_TOKEN
              valueFrom:
                secretKeyRef:
                  key: digitalocean_token
                  name: cloud-credentials
      volumes:
        - name: cloud-credentials
          secret:
            secretName: cloud-credentials
        - name: plugins
          emptyDir: {}
        - name: scratch
          emptyDir: {}

We observe here that we're creating a Deployment called ark that consists of a single replica of the gcr.io/heptio-images/ark:latest container. The Pod is configured using the cloud-credentials secret we created earlier.

Create the Deployment using kubectl apply:

kubectl apply -f examples/20-deployment.yaml

You should see the following output:

Outputdeployment.apps/ark created

We can double check that the Deployment has been successfully created using kubectl get on the heptio-ark Namespace :

kubectl get deployments --namespace=heptio-ark

You should see the following output:

OutputNAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ark       1         1         1            0           8m

The Ark server Pod may not start correctly until you install the Ark DigitalOcean plugin. To install the ark-blockstore-digitalocean plugin, use the ark client we installed earlier:

ark plugin add quay.io/stackpoint/ark-blockstore-digitalocean:latest

You can specify the kubeconfig to use with the --kubeconfig flag. If you don't use this flag, ark will check the KUBECONFIG environment variable and then fall back to the kubectl default (~/.kube/config).

At this point Ark is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.

In the next section, we'll run a quick test to make sure that the backup and restore functionality works as expected.

Step 3 — Testing Backup and Restore Procedure

Now that we've successfully installed and configured Ark, we can create a test Nginx Deployment and Persistent Volume, and run through a backup and restore drill to ensure that everything is working properly.

The ark-plugin-digitalocean repository contains a sample Nginx deployment called nginx-pv.yaml.

Let's take a quick look:

cat examples/nginx-pv.yaml

You should see the following text:

Output---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-logs
  namespace: nginx-example
  labels:
    app: nginx
spec:
  storageClassName: do-block-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:1.7.9
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

In this file, we observe specs for:

An Nginx Deployment consisting of a single replica of the nginx:1.7.9 container image
A 5Gi Persistent Volume Claim (called nginx-logs), using the do-block-storage StorageClass
A LoadBalancer Service that exposes port 80

Create the deployment using kubectl apply:

kubectl apply -f examples/nginx-pv.yml

You should see the following output:

Outputnamespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created

Check that the Deployment succeeded:

kubectl get deployments --namespace=nginx-example

You should see the following output:

OutputNAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1         1         1            1           1h

Once Available reaches 1, fetch the Nginx load balancer’s external IP using kubectl get:

kubectl get services --namespace=nginx-example

You should see both the internal CLUSTER-IP and EXTERNAL-IP for the my-nginx Service:

OutputNAME       TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)        AGE
my-nginx   LoadBalancer   10.32.27.0      203.0.113.0   80:30754/TCP   3m

Note the EXTERNAL-IP and navigate to it using your web browser.

You should see the following NGINX welcome page:

This indicates that your Nginx Deployment and Service are up and running.

Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):

Fetch the Pod’s name using kubectl get:

kubectl get pods --namespace nginx-example

OutputNAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m

Now, exec into the running Nginx container to get a shell inside of it:

kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

cat /var/log/nginx/access.log

You should see some Nginx access entries:

Output10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"
10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure.

We can now perform the backup procedure to copy all nginx Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.

We'll create a backup called nginx-backup using the ark client:

ark backup create nginx-backup --selector app=nginx

The --selector app=nginx instructs the Ark server to only back up Kubernetes objects with the app=nginx Label Selector.

You should see the following output:

OutputBackup request "nginx-backup" submitted successfully.
Run `ark backup describe nginx-backup` for more details.

Running ark backup describe nginx-backup should provide the following output after a short delay:

OutputName:         nginx-backup
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=nginx

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2018-09-26 00:14:30 -0400 EDT
Completed:  2018-09-26 00:14:34 -0400 EDT

Expiration:  2018-10-26 00:14:30 -0400 EDT

Validation errors:  <none>

Persistent Volumes:
  pvc-e4862eac-c2d2-11e8-920b-92c754237aeb:
    Snapshot ID:        2eb66366-c2d3-11e8-963b-0a58ac14428b
    Type:               ext4
    Availability Zone:
    IOPS:               <N/A>

This output indicates that nginx-backup completed successfully.

From the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.

You should see a new directory called nginx-backup containing the Ark backup files.

Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.

We can now test the restore procedure.

Let's first delete the nginx-example Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:

kubectl delete namespace nginx-example

Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example Deployment is no longer running:

kubectl get deployments --namespace=nginx-example

OutputNo resources found.

We can now perform the restore procedure, once again using the ark client:

ark restore create --from-backup nginx-backup

Here we use create to create an Ark Restore object from the nginx-backup object.

You should see the following output:

Output
Restore request "nginx-backup-20180926143828" submitted successfully.
Run `ark restore describe nginx-backup-20180926143828` for more details.

Check the status of the restored Deployment:

kubectl get deployments --namespace=nginx-example

OutputNAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1         1         1            1           1m

Check for the creation of a Persistent Volume:

 kubectl get pvc --namespace=nginx-example

OutputNAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
nginx-logs   Bound     pvc-e4862eac-c2d2-11e8-920b-92c754237aeb   5Gi        RWO            do-block-storage   3m

Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.

Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.

To do this, once again fetch the Pod’s name using kubectl get:

kubectl get pods --namespace nginx-example

OutputNAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m

Then exec into it:

kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

cat /var/log/nginx/access.log

Output10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"
10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.

At this point, we've successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.

Conclusion

In this guide we installed and configured the Ark Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.

Ark can also be used to schedule regular backups of your Kubernetes cluster. To do this, you can use the ark schedule command. It can also be used to migrate resources from one cluster to another. To learn more about these two use cases, consult the official Ark documentation.

To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.

This tutorial builds on the README found in StackPointCloud's ark-plugin-digitalocean GitHub repo.

How To Build a Neural Network to Recognize Handwritten Digits with TensorFlow

2018-09-17T21:58:42Z

Introduction

Neural networks are used as a method of deep learning, one of the many subfields of artificial intelligence. They were first proposed around 70 years ago as an attempt at simulating the way the human brain works, though in a much more simplified form. Individual ‘neurons’ are connected in layers, with weights assigned to determine how the neuron responds when signals are propagated through the network. Previously, neural networks were limited in the number of neurons they were able to simulate, and therefore the complexity of learning they could achieve. But in recent years, due to advancements in hardware development, we have been able to build very deep networks, and train them on enormous datasets to achieve breakthroughs in machine intelligence.

These breakthroughs have allowed machines to match and exceed the capabilities of humans at performing certain tasks. One such task is object recognition. Though machines have historically been unable to match human vision, recent advances in deep learning have made it possible to build neural networks which can recognize objects, faces, text, and even emotions.

In this tutorial, you will implement a small subsection of object recognition—digit recognition. Using TensorFlow, an open-source Python library developed by the Google Brain labs for deep learning research, you will take hand-drawn images of the numbers 0-9 and build and train a neural network to recognize and predict the correct label for the digit displayed.

While you won't need prior experience in practical deep learning or TensorFlow to follow along with this tutorial, we'll assume some familiarity with machine learning terms and concepts such as training and testing, features and labels, optimization, and evaluation. You can learn more about these concepts in An Introduction to Machine Learning.

Prerequisites

To complete this tutorial, you'll need:

A local Python 3 development environment, including pip, a tool for installing Python packages, and venv, for creating virtual environments.

Step 1 — Configuring the Project

Before you can develop the recognition program, you'll need to install a few dependencies and create a workspace to hold your files.

We'll use a Python 3 virtual environment to manage our project's dependencies. Create a new directory for your project and navigate to the new directory:

mkdir tensorflow-demo
cd tensorflow-demo

Execute the following commands to set up the virtual environment for this tutorial:

python3 -m venv tensorflow-demo
source tensorflow-demo/bin/activate

Next, install the libraries you'll use in this tutorial. We'll use specific versions of these libraries by creating a requirements.txt file in the project directory which specifies the requirement and the version we need. Create the requirements.txt file:

touch requirements.txt

Open the file in your text editor and add the following lines to specify the Image, NumPy, and TensorFlow libraries and their versions:

requirements.txt

image==1.5.20
numpy==1.14.3
tensorflow==1.4.0

Save the file and exit the editor. Then install these libraries with the following command:

pip install -r requirements.txt

With the dependencies installed, we can start working on our project.

Step 2 — Importing the MNIST Dataset

The dataset we will be using in this tutorial is called the MNIST dataset, and it is a classic in the machine learning community. This dataset is made up of images of handwritten digits, 28x28 pixels in size. Here are some examples of the digits included in the dataset:

Let's create a Python program to work with this dataset. We will use one file for all of our work in this tutorial. Create a new file called main.py:

touch main.py

Now open this file in your text editor of choice and add this line of code to the file to import the TensorFlow library:

main.py

import tensorflow as tf

Add the following lines of code to your file to import the MNIST dataset and store the image data in the variable mnist:

main.py

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True) # y labels are oh-encoded

When reading in the data, we are using one-hot-encoding to represent the labels (the actual digit drawn, e.g. "3") of the images. One-hot-encoding uses a vector of binary values to represent numeric or categorical values. As our labels are for the digits 0-9, the vector contains ten values, one for each possible digit. One of these values is set to 1, to represent the digit at that index of the vector, and the rest are set to 0. For example, the digit 3 is represented using the vector [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]. As the value at index 3 is stored as 1, the vector therefore represents the digit 3.

To represent the actual images themselves, the 28x28 pixels are flattened into a 1D vector which is 784 pixels in size. Each of the 784 pixels making up the image is stored as a value between 0 and 255. This determines the grayscale of the pixel, as our images are presented in black and white only. So a black pixel is represented by 255, and a white pixel by 0, with the various shades of gray somewhere in between.

We can use the mnist variable to find out the size of the dataset we have just imported. Looking at the num_examples for each of the three subsets, we can determine that the dataset has been split into 55,000 images for training, 5000 for validation, and 10,000 for testing. Add the following lines to your file:

main.py

n_train = mnist.train.num_examples # 55,000
n_validation = mnist.validation.num_examples # 5000
n_test = mnist.test.num_examples # 10,000

Now that we have our data imported, it’s time to think about the neural network.

Step 3 — Defining the Neural Network Architecture

The architecture of the neural network refers to elements such as the number of layers in the network, the number of units in each layer, and how the units are connected between layers. As neural networks are loosely inspired by the workings of the human brain, here the term unit is used to represent what we would biologically think of as a neuron. Like neurons passing signals around the brain, units take some values from previous units as input, perform a computation, and then pass on the new value as output to other units. These units are layered to form the network, starting at a minimum with one layer for inputting values, and one layer to output values. The term hidden layer is used for all of the layers in between the input and output layers, i.e. those "hidden" from the real world.

Different architectures can yield drastically different results, as the performance can be thought of as a function of the architecture among other things, such as the parameters, the data, and the duration of training.

Add the following lines of code to your file to store the number of units per layer in global variables. This allows us to alter the network architecture in one place, and at the end of the tutorial you can test for yourself how different numbers of layers and units will impact the results of our model:

main.py

n_input = 784   # input layer (28x28 pixels)
n_hidden1 = 512 # 1st hidden layer
n_hidden2 = 256 # 2nd hidden layer
n_hidden3 = 128 # 3rd hidden layer
n_output = 10   # output layer (0-9 digits)

The following diagram shows a visualization of the architecture we've designed, with each layer fully connected to the surrounding layers:

The term "deep neural network" relates to the number of hidden layers, with "shallow" usually meaning just one hidden layer, and "deep" referring to multiple hidden layers. Given enough training data, a shallow neural network with a sufficient number of units should theoretically be able to represent any function that a deep neural network can. But it is often more computationally efficient to use a smaller deep neural network to achieve the same task that would require a shallow network with exponentially more hidden units. Shallow neural networks also often encounter overfitting, where the network essentially memorizes the training data that it has seen, and is not able to generalize the knowledge to new data. This is why deep neural networks are more commonly used: the multiple layers between the raw input data and the output label allow the network to learn features at various levels of abstraction, making the network itself better able to generalize.

Other elements of the neural network that need to be defined here are the hyperparameters. Unlike the parameters that will get updated during training, these values are set initially and remain constant throughout the process. In your file, set the following variables and values:

main.py

learning_rate = 1e-4
n_iterations = 1000
batch_size = 128
dropout = 0.5

The learning rate represents ow much the parameters will adjust at each step of the learning process. These adjustments are a key component of training: after each pass through the network we tune the weights slightly to try and reduce the loss. Larger learning rates can converge faster, but also have the potential to overshoot the optimal values as they are updated. The number of iterations refers to how many times we go through the training step, and the batch size refers to how many training examples we are using at each step. The dropout variable represents a threshold at which we elimanate some units at random. We will be using dropout in our final hidden layer to give each unit a 50% chance of being eliminated at every training step. This helps prevent overfitting.

We have now defined the architecture of our neural network, and the hyperparameters that impact the learning process. The next step is to build the network as a TensorFlow graph.

Step 4 — Building the TensorFlow Graph

To build our network, we will set up the network as a computational graph for TensorFlow to execute. The core concept of TensorFlow is the tensor, a data structure similar to an array or list. initialized, manipulated as they are passed through the graph, and updated through the learning process.

We’ll start by defining three tensors as placeholders, which are tensors that we'll feed values into later. Add the following to your file:

main.py

X = tf.placeholder("float", [None, n_input])
Y = tf.placeholder("float", [None, n_output])
keep_prob = tf.placeholder(tf.float32)

The only parameter that needs to be specified at its declaration is the size of the data we will be feeding in. For X we use a shape of [None, 784], where None represents any amount, as we will be feeding in an undefined number of 784-pixel images. The shape of Y is [None, 10] as we will be using it for an undefined number of label outputs, with 10 possible classes. The keep_prob tensor is used to control the dropout rate, and we initialize it as a placeholder rather than an immutable variable because we want to use the same tensor both for training (when dropout is set to 0.5) and testing (when dropout is set to 1.0).

The parameters that the network will update in the training process are the weight and bias values, so for these we need to set an initial value rather than an empty placeholder. These values are essentially where the network does its learning, as they are used in the activation functions of the neurons, representing the strength of the connections between units.

Since the values are optimized during training, we could set them to zero for now. But the initial value actually has a significant impact on the final accuracy of the model. We'll use random values from a truncated normal distribution for the weights. We want them to be close to zero, so they can adjust in either a positive or negative direction, and slightly different, so they generate different errors. This will ensure that the model learns something useful. Add these lines:

main.py

weights = {
    'w1': tf.Variable(tf.truncated_normal([n_input, n_hidden1], stddev=0.1)),
    'w2': tf.Variable(tf.truncated_normal([n_hidden1, n_hidden2], stddev=0.1)),
    'w3': tf.Variable(tf.truncated_normal([n_hidden2, n_hidden3], stddev=0.1)),
    'out': tf.Variable(tf.truncated_normal([n_hidden3, n_output], stddev=0.1)),
}

For the bias, we use a small constant value to ensure that the tensors activate in the intial stages and therefore contribute to the propagation. The weights and bias tensors are stored in dictionary objects for ease of access. Add this code to your file to define the biases:

main.py


biases = {
    'b1': tf.Variable(tf.constant(0.1, shape=[n_hidden1])),
    'b2': tf.Variable(tf.constant(0.1, shape=[n_hidden2])),
    'b3': tf.Variable(tf.constant(0.1, shape=[n_hidden3])),
    'out': tf.Variable(tf.constant(0.1, shape=[n_output]))
}

Next, set up the layers of the network by defining the operations that will manipulate the tensors. Add these lines to your file:

main.py

layer_1 = tf.add(tf.matmul(X, weights['w1']), biases['b1'])
layer_2 = tf.add(tf.matmul(layer_1, weights['w2']), biases['b2'])
layer_3 = tf.add(tf.matmul(layer_2, weights['w3']), biases['b3'])
layer_drop = tf.nn.dropout(layer_3, keep_prob)
output_layer = tf.matmul(layer_3, weights['out']) + biases['out']

Each hidden layer will execute matrix multiplication on the previous layer’s outputs and the current layer’s weights, and add the bias to these values. At the last hidden layer, we will apply a dropout operation using our keep_prob value of 0.5.

The final step in building the graph is to define the loss function that we want to optimize. A popular choice of loss function in TensorFlow programs is cross-entropy, also known as log-loss, which quantifies the difference between two probability distributions (the predictions and the labels). A perfect classification would result in a cross-entropy of 0, with the loss completely minimized.

We also need to choose the optimization algorithm which will be used to minimize the loss function. A process named gradient descent optimization is a common method for finding the (local) minimum of a function by taking iterative steps along the gradient in a negative (descending) direction. There are several choices of gradient descent optimization algorithms already implemented in TensorFlow, and in this tutorial we will be using the Adam optimizer. This extends upon gradient descent optimization by using momentum to speed up the process through computing an exponentially weighted average of the gradients and using that in the adjustments. Add the following code to your file:

main.py

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=Y, logits=output_layer))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

We've now defined the network and built it out with TensorFlow. The next step is to feed data through the graph to train it, and then test that it has actually learnt something.

Step 5 — Training and Testing

The training process involves feeding the training dataset through the graph and optimizing the loss function. Every time the network iterates through a batch of more training images, it updates the parameters to reduce the loss in order to more accurately predict the digits shown. The testing process involves running our testing dataset through the trained graph, and keeping track of the number of images that are correctly predicted, so that we can calculate the accuracy.

Before starting the training process, we will define our method of evaluating the accuracy so we can print it out on mini-batches of data while we train. These printed statements will allow us to check that from the first iteration to the last, loss decreases and accuracy increases; they will also allow us to track whether or not we have ran enough iterations to reach a consistent and optimal result:

main.py

correct_pred = tf.equal(tf.argmax(output_layer, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

In correct_pred, we use the arg_max function to compare which images are being predicted correctly by looking at the output_layer (predictions) and Y (labels), and we use the equal function to return this as a list of [Booleans](tps://www.digitalocean.com/community/tutorials/understanding-data-types-in-python-3#booleans). We can then cast this list to floats and calculate the mean to get a total accuracy score.

We are now ready to initialize a session for running the graph. In this session we will feed the network with our training examples, and once trained, we feed the same graph with new test examples to determine the accuracy of the model. Add the following lines of code to your file:

main.py

init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)

The essence of the training process in deep learning is to optimize the loss function. Here we are aiming to minimize the difference between the predicted labels of the images, and the true labels of the images. The process involves four steps which are repeated for a set number of iterations:

Propagate values forward through the network
Compute the loss
Propagate values backward through the network
Update the parameters

At each training step, the parameters are adjusted slightly to try and reduce the loss for the next step. As the learning progresses, we should see a reduction in loss, and eventually we can stop training and use the network as a model for testing our new data.

Add this code to the file:

main.py

# train on mini batches
for i in range(n_iterations):
    batch_x, batch_y = mnist.train.next_batch(batch_size)
    sess.run(train_step, feed_dict={X: batch_x, Y: batch_y, keep_prob:dropout})

    # print loss and accuracy (per minibatch)
    if i%100==0:
        minibatch_loss, minibatch_accuracy = sess.run([cross_entropy, accuracy], feed_dict={X: batch_x, Y: batch_y, keep_prob:1.0})
        print("Iteration", str(i), "\t| Loss =", str(minibatch_loss), "\t| Accuracy =", str(minibatch_accuracy))

After 100 iterations of each training step in which we feed a mini-batch of images through the network, we print out the loss and accuracy of that batch. Note that we should not be expecting a decreasing loss and increasing accuracy here, as the values are per batch, not for the entire model. We use mini-batches of images rather than feeding them through individually to speed up the training process and allow the network to see a number of different examples before updating the parameters.

Once the training is complete, we can run the session on the test images. This time we are using a keep_prob dropout rate of 1.0 to ensure all units are active in the testing process.

Add this code to the file:

main.py

test_accuracy = sess.run(accuracy, feed_dict={X: mnist.test.images, Y: mnist.test.labels, keep_prob:1.0})
print("\nAccuracy on test set:", test_accuracy)

It’s now time to run our program and see how accurately our neural network can recognize these handwritten digits. Save the main.py file and execute the following command in the terminal to run the script:

python3 main.py

You'll see an output similar to the following, although individual loss and accuracy results may vary slightly:

OutputIteration 0     | Loss = 3.67079    | Accuracy = 0.140625
Iteration 100   | Loss = 0.492122   | Accuracy = 0.84375
Iteration 200   | Loss = 0.421595   | Accuracy = 0.882812
Iteration 300   | Loss = 0.307726   | Accuracy = 0.921875
Iteration 400   | Loss = 0.392948   | Accuracy = 0.882812
Iteration 500   | Loss = 0.371461   | Accuracy = 0.90625
Iteration 600   | Loss = 0.378425   | Accuracy = 0.882812
Iteration 700   | Loss = 0.338605   | Accuracy = 0.914062
Iteration 800   | Loss = 0.379697   | Accuracy = 0.875
Iteration 900   | Loss = 0.444303   | Accuracy = 0.90625

Accuracy on test set: 0.9206

To try and improve the accuracy of our model, or to learn more about the impact of tuning hyperparameters, we can test the effect of changing the learning rate, the dropout threshold, the batch size, and the number of iterations. We can also change the number of units in our hidden layers, and change the amount of hidden layers themselves, to see how different architectures increase or decrease the model accuracy.

To demonstrate that the network is actually recognizing the hand-drawn images, let's test it on a single image of our own.

First either download this sample test image or open up a graphics editor and create your own 28x28 pixel image of a digit.

Open the main.py file in your editor and add the following lines of code to the top of the file to import two libraries necessary for image manipulation.

main.py

import numpy as np
from PIL import Image
...

Then at the end of the file, add the following line of code to load the test image of the handwritten digit:

main.py

img = np.invert(Image.open("test_img.png").convert('L')).ravel()

The open function of the Image library loads the test image as a 4D array containing the three RGB color channels and the Alpha transparency. This is not the same representation we used previously when reading in the dataset with TensorFlow, so we'll need to do some extra work to match the format.

First, we use the convert function with the L parameter to reduce the 4D RGBA representation to one grayscale color channel. We store this as a numpy array and invert it using np.invert, because the current matrix represents black as 0 and white as 255, whereas we need the opposite. Finally, we call ravel to flatten the array.

Now that the image data is structured correctly, we can run a session in the same way as previously, but this time only feeding in the single image for testing. Add the following code to your file to test the image and print the outputted label.

main.py

prediction = sess.run(tf.argmax(output_layer,1), feed_dict={X: [img]})
print ("Prediction for test image:", np.squeeze(prediction))

The np.squeeze function is called on the prediction to return the single integer from the array (i.e. to go from [2] to 2). The resulting output demonstrates that the network has recognized this image as the digit 2.

OutputPrediction for test image: 2

You can try testing the network with more complex images –– digits that look like other digits, for example, or digits that have been drawn poorly or incorrectly –– to see how well it fares.

Conclusion

In this tutorial you successfully trained a neural network to classify the MNIST dataset with around 92% accuracy and tested it on an image of your own. Current state-of-the-art research achieves around 99% on this same problem, using more complex network architectures involving convolutional layers. These use the 2D structure of the image to better represent the contents, unlike our method which flattened all the pixels into one vector of 784 units. You can read more about this topic on the TensorFlow website, and see the research papers detailing the most accurate results on the MNIST website.

Now that you know how to build and train a neural network, you can try and use this implementation on your own data, or test it on other popular datasets such as the Google StreetView House Numbers, or the CIFAR-10 dataset for more general image recognition.

How to Speed Up WordPress Asset Delivery Using DigitalOcean Spaces CDN

2018-08-02T21:06:47Z

Introduction

Implementing a CDN, or Content Delivery Network, to deliver your WordPress site's static assets can greatly decrease your servers' bandwidth usage as well as speed up page load times for geographically dispersed users. WordPress static assets include images, CSS stylesheets, and JavaScript files. Leveraging a system of edge servers distributed worldwide, a CDN caches copies of your site's static assets across its network to reduce the distance between end users and this bandwidth-intensive content.

In a previous Solutions guide, How to Store WordPress Assets on DigitalOcean Spaces, we covered offloading a WordPress site's Media Library (where images and other site content gets stored) to DigitalOcean Spaces, a highly redundant object storage service. We did this using the DigitalOcean Spaces Sync plugin, which automatically syncs WordPress uploads to your Space, allowing you to delete these files from your server and free up disk space.

In this Solutions guide, we'll extend this procedure by rewriting Media Library asset URLs. This forces users’ browsers to download static assets directly from the DigitalOcean Spaces CDN, a geographically distributed set of cache servers optimized for delivering static content. We’ll go over how to enable the CDN for Spaces, how to rewrite links to serve your WordPress assets from the CDN, and finally how to test that your website’s assets are being correctly delivered by the CDN.

Additionally, we’ll demonstrate how to implement Media Library offload and link rewriting using two popular paid WordPress plugins: WP Offload Media and Media Library Folders Pro. You should choose the plugin that suits your production needs best.

Prerequisites

Before you begin this tutorial, you should have a running WordPress installation on top of a LAMP or LEMP stack. You should also have WP-CLI installed on your WordPress server, which you can learn to set up by following these instructions.

To offload your Media Library, you’ll need a DigitalOcean Space and an access key pair:

To learn how to create a Space, consult the Spaces product documentation.
To learn how to create an access key pair and upload files to your Space using the open source s3cmd tool, consult s3cmd 2.x Setup, also on the DigitalOcean product documentation site.

There are a few WordPress plugins that you can use to offload your WordPress assets:

DigitalOcean Spaces Sync is a free and open-source WordPress plugin for offloading your Media Library to a DigitalOcean Space. You can learn how to do this in How To Store WordPress Assets on DigitalOcean Spaces.
WP Offload Media is a paid plugin that copies files from your WordPress Media Library to DigitalOcean Spaces and rewrites URLs to serve the files from the CDN. With the Assets Pull addon, it can identify assets (CSS, JS, images, etc) used by your site (for example by WordPress themes) and also serve these from CDN.
Media Library Folders Pro is another paid plugin that helps you organize your Media Library assets, as well as offload them to DigitalOcean Spaces.

For testing purposes, be sure to have a modern web browser such as Google Chrome or Firefox installed on your client (e.g. laptop) computer.

Once you have a running WordPress installation and have created a DigitalOcean Space, you're ready to enable the CDN for your Space and begin with this guide.

Enabling Spaces CDN

We’ll begin this guide by enabling the CDN for your DigitalOcean Space. This will not affect the availability of existing objects. With the CDN enabled, objects in your Space will be “pushed out” to edge caches across the content delivery network, and a new CDN endpoint URL will be made available to you. To learn more about how CDNs work, consult Using a CDN to Speed Up Static Content Delivery.

First, enable the CDN for your Space by following How to Enable the Spaces CDN.

Navigate back to your Space and reload the page. You should see a new Endpoints link under your Space name:

These endpoints should contain your Space name. We’re using wordpress-offload in this tutorial.

Notice the addition of the new Edge endpoint. This endpoint routes requests for Spaces objects through the CDN, serving them from the edge cache as much as possible. Note down this Edge endpoint, which you’ll use to configure your WordPress plugin in future steps.

Now that you have enabled the CDN for your Space, you’re ready to begin configuring your asset offload and link rewriting plugin.

If you’re using DigitalOcean Spaces Sync and continuing from How to Store WordPress Assets on DigitalOcean Spaces, begin reading from the following section. If you’re not using Spaces Sync, skip to either the WP Offload Media section or the Media Library Folders Pro section, depending on the plugin you choose to use.

Spaces Sync Plugin

If you’d like to use the free and open-source DigitalOcean Spaces Sync and CDN Enabler plugins to serve your files from the CDN's edge caches, follow the steps outlined in this section.

We’ll begin by ensuring that our WordPress installation and Spaces Sync plugin are configured correctly and are serving assets from DigitalOcean Spaces.

Modifying Spaces Sync Plugin Configuration

Continuing from How To Store WordPress Assets on DigitalOcean Spaces, your Media Library should be offloaded to your DigitalOcean Space and your Spaces Sync plugin settings should look as follows:

We are going to make some minor changes to ensure that our configuration allows us to offload WordPress themes and other directories, beyond the wp-content/uploads Media Library folder.

First, we're going to modify the Full URL-path to files field so that the Media Library files are served from our Space’s CDN and not locally from the server. This setting essentially rewrites links to Media Library assets, changing them from file links hosted locally on your WordPress server, to file links hosted on the DigitalOcean Spaces CDN.

Recall the Edge endpoint you noted down in the Enabling Spaces CDN step.

In this tutorial, the Space's name is wordpress-offload and the Space's CDN endpoint is:

https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com

Now, in the Spaces Sync plugin settings page, replace the URL in the Full URL-path to files field with your Spaces CDN endpoint, followed by /wp-content/uploads.

In this tutorial, using the above Spaces CDN endpoint, the full URL would be:

https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com/wp-content/uploads

Next, for the Local path field, enter the full path to the wp-content/uploads directory on your WordPress server. In this tutorial, the path to the WordPress installation on the server is /var/www/html/, so the full path to uploads would be /var/www/html/wp-content/uploads.

Note: If you’re continuing from How To Store WordPress Assets on DigitalOcean Spaces, this guide will slightly modify the path to files in your Space to enable you to optionally offload themes and other wp-content assets. You should clear out your Space before doing this, or alternatively you can transfer existing files into the correct wp-content/uploads Space directory using s3cmd.

In the Storage prefix field, we're going to enter /wp-content/uploads, which will ensure that we build the correct wp-content directory hierarchy so that we can offload other WordPress directories to this Space.

Filemask can remain wildcarded with *, unless you'd like to exclude certain files.

It's not necessary to check the Store files only in the cloud and delete... option; only check this box if you'd like to delete the Media Library assets from your server after they've been successfully uploaded to your DigitalOcean Space.

Your final settings should look something like this:

Be sure to replace the above values with the values corresponding to your WordPress installation and Spaces configuration.

Finally, hit Save Changes.

You should see a Settings saved box appear at the top of your screen, confirming that the Spaces Sync plugin settings have successfully been updated.

Future WordPress Media Library uploads should now be synced to your DigitalOcean Space, and served using the Spaces Content Delivery Network.

In this step, we did not offload the WordPress theme or other wp-content assets. To learn how to transfer these assets to Spaces and serve them using the Spaces CDN, skip to Offload Additional Assets.

To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Test CDN Caching.

WordPress Offload Media Plugin

The DeliciousBrains WordPress Offload Media plugin allows you to quickly and automatically upload your Media Library assets to DigitalOcean Spaces and rewrite links to these assets so that you can deliver them directly from Spaces or via the Spaces CDN. In addition, the Assets Pull addon allows you to quickly offload additional WordPress assets like JS, CSS, and font files in combination with a pull CDN. Setting up this addon is beyond the scope of this guide but to learn more you can consult the DeliciousBrains documentation.

We'll begin by installing and configuring the WP Offload Media plugin for a sample WordPress site.

Installing WP Offload Media Plugin

To begin, you must purchase a copy of the plugin on the DeliciousBrains plugin site. Choose the appropriate version depending on the number of assets in your Media Library, and support and feature requirements for your site.

After going through checkout, you'll be brought to a post-purchase site with a download link for the plugin and a license key. The download link and license key will also be sent to you at the email address you provided when purchasing the plugin.

Download the plugin and navigate to your WordPress site's admin interface (https://your_site_url/wp-admin). Log in if necessary. From here, hover over Plugins and click on Add New.

Click Upload Plugin and the top of the page, Choose File, and then select the zip archive you just downloaded.

Click Install Now, and then Activate Plugin. You'll be brought to WordPress’s plugin admin interface.

From here, navigate to the WP Offload Media plugin's settings page by clicking Settings under the plugin name.

You'll be brought to the following screen:

Click the radio button next to DigitalOcean Spaces. You'll now be prompted to either configure your Spaces Access Key in the wp-config.php file (recommended), or directly in the web interface (the latter will store your Spaces credentials in the WordPress database).

We'll configure our Spaces Access Key in wp-config.php.

Log in to your WordPress server via the command line, and navigate to your WordPress root directory (in this tutorial, this is /var/www/html). From here, open up wp-config.php in your favorite editor:

sudo nano wp-config.php

Scroll down to the line that says /* That's all, stop editing! Happy blogging. */, and before it insert the following lines containing your Spaces Access Key pair (to learn how to generate an access key pair, consult the Spaces product docs):

wp-config.php

. . . 
define( 'AS3CF_SETTINGS', serialize( array(
    'provider' => 'do',
    'access-key-id' => 'your_access_key_here',
    'secret-access-key' => 'your_secret_key_here',
) ) );

/* That's all, stop editing! Happy blogging. */
. . .

Once you're done editing, save and close the file. The changes will take effect immediately.

Back in the WordPress Offload Media plugin admin interface, select the radio button next to Define access keys in wp-config.php and hit Save Changes.

You should be brought to the following interface:

On this configuration page, select the appropriate region for your Space using the Region dropdown and enter your Space name next to Bucket (in this tutorial, our Space is called wordpress-offload).

Then, hit Save Bucket.

You'll be brought to the main WP Offload Media configuration page. At the top you should see the following warning box:

Click on enter your license key, and on the subsequent page enter the license key found in your email receipt or on the checkout page and hit Activate License.

If you entered your license key correctly, you should see License activated successfully.

Now, navigate back to main WP Offload Media configuration page by clicking on Media Library at the top of the window.

At this point, WP Offload Media has successfully been configured for use with your DigitalOcean Space. You can now begin offloading assets and delivering them using the Spaces CDN.

Configuring WP Offload Media

Now that you've linked WP Offload Media with your DigitalOcean Space, you can begin offloading assets and configuring URL rewriting to deliver media from the Spaces CDN.

You should see the following configuration options on the main WP Offload Media configuration page:

These defaults should work fine for most use cases. If your Media Library exists at a nonstandard path within your WordPress directory, enter the path in the text box under the Path option.

If you'd like to change asset URLs so that they are served directly from Spaces and not your WordPress server, ensure the toggle is set to On next to Rewrite Media URLs.

To deliver Media Library assets using the Spaces CDN, ensure you've enabled the CDN for your Space (see Enable Spaces CDN to learn how) and have noted down the URL for the Edge endpoint. Hit the toggle next to Custom Domain (CNAME), and In the text box that appears, enter the CDN Edge endpoint URL, without the https:// prefix.

In this guide the Spaces CDN endpoint is:

https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com

So here we enter:

 wordpress-offload.nyc3.cdn.digitaloceanspaces.com

To improve security, we'll force HTTPS for requests to Media Library assets (now served using the CDN) by setting the toggle to On.

You can optionally clear out files that have been offloaded to Spaces from your WordPress server to free up disk space. To do this, hit On next to Remove Files From Server.

Once you've finished configuring WP Offload Media, hit Save Changes at the bottom of the page to save your settings.

The URL Preview box should display a URL containing your Spaces CDN endpoint. It should look something like the following:

https://wordpress‑offload.nyc3.cdn.digitaloceanspaces.com/wp‑content/uploads/2018/09/21211354/photo.jpg

This URL indicates that WP Offload Media has been successfully configured to deliver Media Library assets using the Spaces CDN. If the path doesn't contain cdn, ensure that you correctly entered the Edge endpoint URL and not the Origin URL.

At this point, WP Offload Media has been set up to deliver your Media Library using Spaces CDN. Any future uploads to your Media Library will be automatically copied over to your DigitalOcean Space and served using the CDN.

You can now bulk offload existing assets in your Media Library using the built-in upload tool.

Offloading Media Library

We'll use the plugin's built-in "Upload Tool" to offload existing files in our WordPress Media Library.

On the right-hand side of the main WP Offload Media configuration page, you should see the following box:

Click Offload Now to upload your Media Library files to your DigitalOcean Space.

If the upload procedure gets interrupted, the box will change to display the following:

Hit Offload Remaining Now to transfer the remaining files to your DigitalOcean Space.

Once you've offloaded the remaining items from your Media Library, you should see the following new boxes:

At this point you've offloaded your Media Library to your Space and are delivering the files to users using the Spaces CDN.

At any point in time, you can download the files back to your WordPress server from your Space by hitting Download Files.

You can also clear out your DigitalOcean Space by hitting Remove Files. Before doing this, ensure that you’ve first downloaded the files back to your WordPress server from Spaces.

In this step, we learned how to offload our WordPress Media Library to DigitalOcean Spaces and rewrite links to these Library assets using the WP Offload Media plugin.

To offload additional WordPress assets like themes and JavaScript files, you can use the Asset Pull addon or consult the Offload Additional Assets section of this guide.

To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Test CDN Caching.

Media Library Folders Pro and CDN Enabler Plugins

The MaxGalleria Media Library Folders Pro plugin is a convenient WordPress plugin that allows you to better organize your WordPress Media Library assets. In addition, the free Spaces addon allows you to bulk offload your Media Library assets to DigitalOcean Spaces, and rewrite URLs to those assets to serve them directly from object storage. You can then enable the Spaces CDN and use the Spaces CDN endpoint to serve your library assets from the distributed delivery network. To accomplish this last step, you can use the CDN Enabler plugin to rewrite CDN endpoint URLs for your Media Library assets.

We'll begin by installing and configuring the Media Library Folders Pro (MLFP) plugin, as well as the MLFP Spaces addon. We’ll then install and configure the CDN Enabler plugin to deliver Media Library assets using the Spaces CDN.

Installing MLFP Plugin

After purchasing the MLFP plugin, you should have received an email containing your MaxGalleria account credentials as well as a plugin download link. Click on the plugin download link to download the MLFP plugin zip archive to your local computer.

Once you've downloaded the archive, log in to your WordPress site's administration interface (https://your_site_url/wp-admin), and navigate to Plugins and then Add New in the left-hand sidebar.

From the Add Plugins page, click Upload Plugin and then select the zip archive you just downloaded.

Click Install Now to complete the plugin installation, and from the Installing Plugin screen, click Activate Plugin to activate MLFP.

You should then see a Media Library Folders Pro menu item appear in the left-hand sidebar. Click it to go to the Media Library Folders Pro interface. Covering the plugin's various features is beyond the scope of this guide, but to learn more, you can consult the MaxGalleria site and forums.

We'll now activate the plugin. Click into Settings under the MLFP menu item, and enter your license key next to the License Key text box. You can find your MLFP license key in the email sent to you when you purchased the plugin. Hit Save Changes and then Activate License. Next, hit Update Settings.

Your MLFP plugin is now active, and you can use it to organize existing or new Media Library assets for your WordPress site.

We'll now install and configure the Spaces addon plugin so that you can offload and serve these assets from DigitalOcean Spaces.

Installing MLFP Spaces Addon Plugin and Offload Media Library

To install the Spaces Addon, log in to your MaxGalleria account. You can find your account credentials in an email sent to you when you purchased the MLFP plugin.

Navigate to the Addons page in the top menu bar and scroll down to Media Sources. From here, click into the Media Library Folders Pro S3 and Spaces option.

From this page, scroll down to the Pricing section and select the option that suits the size of your WordPress Media Library (for Media Libraries with 3000 images or less, the addon is free).

After completing the addon "purchase," you can navigate back to your account page (by clicking the Account link in the top menu bar), from which the addon plugin will now be available.

Click on the Media Library Folders Pro S3 image and the plugin download should begin.

Once the download completes, navigate back to your WordPress administration interface, and install the downloaded plugin using the same method as above, by clicking Upload Plugin. Once again, hit Activate Plugin to activate the plugin.

You will likely receive a warning about configuring access keys in your wp-config.php file. We'll configure these now.

Log in to your WordPress server using the console or SSH, and navigate to your WordPress root directory (in this tutorial, this is /var/www/html). From here, open up wp-config.php in your favorite editor:

sudo nano wp-config.php

Scroll down to the line that says /* That's all, stop editing! Happy blogging. */, and before it insert the following lines containing your Spaces Access Key pair and a plugin configuration option (to learn how to generate an access key pair, consult the Spaces product docs):

wp-config.php

. . . 
define('MF_AWS_ACCESS_KEY_ID', 'your_access_key_here');
define( 'MF_AWS_SECRET_ACCESS_KEY', 'your_secret_key_here');
define('MF_CLOUD_TYPE', 'do')

/* That's all, stop editing! Happy blogging. */
. . .

Once you're done editing, save and close the file.

Now, navigate to your DigitalOcean Space from the Cloud Control Panel, and create a folder called wp-content by clicking on New Folder.

From here, navigate back to the WordPress administration interface, and click into Media Library Folders Pro and then S3 & Spaces Settings in the sidebar.

The warning banner about configuring access keys should now have disappeared. If it's still present, you should double check your wp-config.php file for any typos or syntax errors.

In the License Key text box, enter the license key that was emailed to you after purchasing the Spaces addon. Note that this license key is different from the MLFP license key. Hit Save Changes and then Activate License.

Once activated, you should see the following configuration pane:

From here, click Select Image Bucket & Region to select your DigitalOcean Space. Then select the correct region for your Space and hit Save Bucket Selection.

You've now successfully connected the Spaces offload plugin to your DigitalOcean Space. You can begin offloading your WordPress Media Library assets.

The Use files on the cloud server checkbox allows you to specify where Media Library assets will be served from. If you check the box, assets will be served from DigitalOcean Spaces, and URLs to images and other Media Library objects will be correspondingly rewritten. If you plan on using the Spaces CDN to serve your Media Library assets, do not check this box, as the plugin will use the Spaces Origin endpoint and not the CDN Edge endpoint. We will configure CDN link rewriting in a future step.

Click the Remove files from local server box to delete local Media Library assets once they've been successfully uploaded to DigitalOcean Spaces.

The Remove individual downloaded files from the cloud server checkbox should be used when bulk downloading files from Spaces to your WordPress server. If checked, these files will be deleted from Spaces after successfully downloading to your WordPress server. We can ignore this option for now.

Since we're configuring the plugin for use with the Spaces CDN, leave the Use files on the cloud server box unchecked, and hit Copy Media Library to the cloud server to sync your site's WordPress Media Library to your DigitalOcean Space.

You should see a progress box appear, and then Upload complete. indicating the Media Library sync has concluded successfully.

Navigate to your DigitalOcean Space to confirm that your Media Library files have been copied to your Space. They should be available in the uploads subdirectory of the wp-content directory you created earlier in this step.

Once your files are available in your Space, you're ready to move on to configuring the Spaces CDN.

Installing CDN Enabler Plugin to Deliver Assets from Spaces CDN

To use the Spaces CDN to serve your now offloaded files, first ensure that you've enabled the CDN for your Space.

Once the CDN has been enabled for your Space, you can now install and configure the CDN Enabler WordPress plugin to rewrite links to your Media Library assets. The plugin will rewrite links to these assets so that they are served from the Spaces CDN endpoint.

To install CDN Enabler, you can either use the Plugins menu from the WordPress administration interface, or install the plugin directly from the command line. We'll demonstrate the latter procedure here.

First, log in to your WordPress server. Then, navigate to your plugins directory:

cd /var/www/html/wp-content/plugins

Be sure to replace the above path with the path to your WordPress installation.

From the command line, use the wp-cli interface to install the plugin:

wp plugin install cdn-enabler

Now, activate the plugin:

wp plugin activate cdn-enabler

Back in the WordPress Admin Area, under Settings, you should see a new link to CDN Enabler settings. Click into CDN Enabler.

You should see the following settings screen:

Modify the displayed fields as follows:

CDN URL: Enter the Spaces Edge endpoint, which you can find from the Spaces Dashboard. In this tutorial, this is https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
Included Directories: Enter wp-content/uploads. We'll learn how to serve other wp-content directories in the Offload Additional Assets section.
Exclusions: Leave the default .php
Relative Path: Leave the box checked
CDN HTTPS: Enable it by checking the box
Leave the remaining two fields blank

Then, hit Save Changes to save these settings and enable them for your WordPress site.

At this point you've successfully offloaded your WordPress site's Media Library to DigitalOcean Spaces and are serving them to end users using the CDN.

To verify and test that your Media Library uploads are being delivered from the Spaces CDN, skip to Test CDN Caching.

Offloading Additional Assets (Optional)

In previous sections of this guide, we’ve learned how to offload our site’s WordPress Media Library to Spaces and serve these files using the Spaces CDN. In this section, we’ll cover offloading and serving additional WordPress assets like themes, JavaScript files, and fonts.

Most of these static assets live inside of the wp-content directory (which contains wp-themes). To offload and rewrite URLs for this directory, we’ll use CDN Enabler, an open-source plugin developed by KeyCDN.

If you’re using the WP Offload Media plugin, you can use the Asset Pull addon to serve these files using a pull CDN. Installing and configuring this addon is beyond the scope of this guide. To learn more, consult the DeliciousBrains product page.

First, we’ll install CDN Enabler. We’ll then copy our WordPress themes over to Spaces, and finally configure CDN Enabler to deliver these using the Spaces CDN.

If you’ve already installed CDN Enabler in a previous step, skip to Step 2.

Step 1 — Installing CDN Enabler

To install CDN Enabler, log in to your WordPress server. Then, navigate to your plugins directory:

cd /var/www/html/wp-content/plugins

Be sure to replace the above path with the path to your WordPress installation.

From the command line, use the wp-cli interface to install the plugin:

wp plugin install cdn-enabler

Now, activate the plugin:

wp plugin activate cdn-enabler

Back in the WordPress Admin Area, under Settings, you should see a new link to CDN Enabler settings. Click into CDN Enabler.

You should see the following settings screen:

At this point you’ve successfully installed CDN Enabler. We’ll now upload our WordPress themes to Spaces.

Step 2 — Uploading Static WordPress Assets to Spaces

In this tutorial, to demonstrate a basic plugin configuration, we're only going to serve wp-content/themes, the WordPress directory containing WordPress themes' PHP, JavaScript, HTML, and image files. You can optionally extend this process to other WordPress directories, like wp-includes, and even the entire wp-content directory.

The theme used by the WordPress installation in this tutorial is twentyseventeen, the default theme for a fresh WordPress installation at the time of writing. You can repeat these steps for any other theme or WordPress content.

First, we'll upload our theme to our DigitalOcean Space using s3cmd. If you haven't yet configured s3cmd, consult the DigitalOcean Spaces Product Documentation.

Navigate to your WordPress installation's wp-content directory:

cd /var/www/html/wp-content

From here, upload the themes directory to your DigitalOcean Space using s3cmd. Note that at this point you can choose to upload only a single theme, but for simplicity and to offload as much content as possible from our server, we will upload all the themes in the themes directory to our Space.

We'll use find to build a list of non-PHP (therefore cacheable) files, which we'll then pipe to s3cmd to upload to Spaces. We’ll exclude CSS stylesheets as well in this first command as we need to set the text/css MIME type when uploading them.

find themes/ -type f -not \( -name '*.php' -or -name '*.css' \) | xargs -I{} s3cmd put --acl-public {} s3://wordpress-offload/wp-content/{}

Here, we instruct find to search for files within the themes/ directory, and ignore .php and .css files. We then use xargs -I{} to iterate over this list, executing s3cmd put for each file, and set the file's permissions in Spaces to public using --acl-public.

Next, we’ll do the same for CSS stylesheets, adding the --mime-type="text/css" flag to set the text/css MIME type for the stylesheets on Spaces. This will ensure that Spaces serves your theme's CSS files using the correct Content-Type: text/css HTTP header:

find themes/ -type f -name '*.css' | xargs -I{} s3cmd put --acl-public --mime-type="text/css" {} s3://wordpress-offload/wp-content/{}

Again, be sure to replace wordpress-offload in the above command with your Space name.

Now that we’ve uploaded our theme, let’s verify that it can be found at the correct path in our Space. Navigate to your Space using the DigitalOcean Cloud Control Panel.

Enter the wp-content directory, followed by the themes directory. You should see your theme's directory here. If you don't, verify your s3cmd configuration and re-upload your theme to your Space.

Step 3 — Configuring CDN Enabler to Rewrite Asset Links

Now that our theme lives in our Space, and we've set the correct metadata, we can begin serving its files using CDN Enabler and the DigitalOcean Spaces CDN.

Navigate back to the WordPress Admin Area and click into Settings and then CDN Enabler.

Here, modify the displayed fields as follows:

CDN URL: Enter the Spaces Edge endpoint, as done in Step 1. In this tutorial, this is https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com
Included Directories: If you’re not using the MLFP plugin, this should be wp-content/themes. If you are, this should be wp-content/uploads,wp-content/themes
Exclusions: Leave the default .php
Relative Path: Leave the box checked
CDN HTTPS: Enable it by checking the box
Leave the remaining two fields blank

Your final settings should look something like this:

Hit Save Changes to save these settings and enable them for your WordPress site.

At this point you've successfully offloaded your WordPress site's theme assets to DigitalOcean Spaces and are serving them to end users using the CDN. We can confirm this using Chrome's DevTools, following the procedure described below.

Using the CDN Enabler plugin, you can repeat this process for other WordPress directories, like wp-includes, and even the entire wp-content directory.

Testing CDN Caching

In this section, we’ll demonstrate how to determine where your WordPress assets are being served from (e.g. your host server or the CDN) using Google Chrome’s DevTools.

Step 1 — Adding Sample Image to Media Library to Test Syncing

To begin, we’ll first upload a sample image to our Media Library, and verify that it's being served from the DigitalOcean Spaces CDN servers. You can upload an image using the WordPress Admin web interface, or using the wp-cli command-line tool. In this guide, we’ll use wp-cli to upload the sample image.

Log in to your WordPress server using the command line, and navigate to the home directory for the non-root user you've configured. In this tutorial, we’ll use the user sammy.

cd

From here, use curl to download the DigitalOcean logo to your Droplet (if you already have an image you'd like to test with, skip this step):

curl https://assets.digitalocean.com/logos/DO_Logo_horizontal_blue.png > do_logo.png

Now, use wp-cli to import the image to your Media Library:

wp media import --path=/var/www/html/ /home/sammy/do_logo.png

Be sure to replace /var/www/html with the correct path to the directory containing your WordPress files.

You may see some warnings, but the output should end in the following:

OutputImported file '/home/sammy/do_logo.png' as attachment ID 10.
Success: Imported 1 of 1 items.

Which indicates that our test image has successfully been copied to the WordPress Media Library, and also uploaded to our DigitalOcean Space, using your preferred offload plugin.

Navigate to your DigitalOcean Space to confirm:

This indicates that your offload plugin is functioning as expected and automatically syncing WordPress uploads to your DigitalOcean Space. Note that the exact path to your Media Library uploads in the Space will depend on the plugin you’re using to offload your WordPress files.

Next, we will verify that this file is being served using the Spaces CDN, and not from the server running WordPress.

Step 2 — Inspecting Asset URL

From the WordPress admin area (https://your_domain/wp-admin), navigate to Pages in the left-hand side navigation menu.

We will create a sample page containing our uploaded image to determine where it's being served from. You can also run this test by adding the image to an existing page on your WordPress site.

From the Pages screen, click into Sample Page, or any existing page. You can alternatively create a new page.

In the page editor, click on Add Media, and select the DigitalOcean logo (or other image you used to test this procedure).

An Attachment Details pane should appear on the right-hand side of your screen. From this pane, add the image to the page by clicking on Insert into page.

Now, back in the page editor, click on either Publish (if you created a new sample page) or Update (if you added the image to an existing page) in the Publish box on the right-hand side of your screen.

Now that the page has successfully been updated to contain the image, navigate to it by clicking on the Permalink under the page title. You'll be brought to this page in your web browser.

For the purposes of this tutorial, the following steps will assume that you're using Google Chrome, but you can use most modern web browsers to run a similar test.

From the rendered page preview in your browser, right click on the image and click on Inspect:

A DevTools window should pop up, highlighting the img asset in the page's HTML:

You should see the CDN endpoint for your DigitalOcean Space in this URL (in this tutorial, our Spaces CDN endpoint is https://wordpress-offload.nyc3.cdn.digitaloceanspaces.com), indicating that the image asset is being served from the DigitalOcean Spaces CDN edge cache.

This confirms that your Media Library uploads are being synced to your DigitalOcean Space and served using the Spaces CDN.

Step 3 — Inspecting Asset Response Headers

From the DevTools window, we'll run one final test. Click on Network in the toolbar at the top of the window.

Once in the blank Network window, follow the displayed instructions to reload the page.

The page assets should populate in the window. Locate your test image in the list of page assets:

Once you've located your test image, click into it to open an additional information pane. Within this pane, click on Headers to show the response headers for this asset:

You should see the Cache-Control HTTP header, which is a CDN response header. This confirms that this image was served from the Spaces CDN.

Step 4 — Inspecting URLs for Theme Assets (Optional)

If you offloaded your wp-themes (or other) directory as described in Offload Additional Assets, you should perform the following brief check to verify that your theme’s assets are being served from the Spaces CDN.

Navigate to your WordPress site in Google Chrome, and right-click anywhere in the page. In the menu that appears, click on Inspect.

You'll once again be brought to the Chrome DevTools interface.

From here, click into Sources.

In the left-hand pane, you should see a list of your WordPress site's assets. Scroll down to your CDN endpoint, and expand the list by clicking the small arrow next to the endpoint name:

Observe that your WordPress theme's header image, JavaScript, and CSS stylesheet are now being served from the Spaces CDN.

Conclusion

In this tutorial, we've shown how to offload static content from your WordPress server to DigitalOcean Spaces, and serve this content using the Spaces CDN. In most cases, this should reduce bandwidth on your host infrastructure and speed up page loads for end users, especially those located further away geographically from your WordPress server.

We demonstrated how to offload and serve both Media Library and themes assets using the Spaces CDN, but these steps can be extended to further unload the entire wp-content directory, as well as wp-includes.

Implementing a CDN to deliver static assets is just one way to optimize your WordPress installation. Other plugins like W3 Total Cache can further speed up page loads and improve the SEO of your site. A helpful tool to measure your page load speed and improve it is Google's PageSpeed Insights. Another helpful tool that provides a waterfall breakdown of request and response times as well as suggested optimizations is Pingdom.

To learn more about Content Delivery Networks and how they work, consult Using a CDN to Speed Up Static Content Delivery.

How To Manage an SQL Database

2018-09-26T14:34:52Z

An SQL Cheat Sheet

Introduction

SQL databases come installed with all the commands you need to add, modify, delete, and query your data. This cheat sheet-style guide provides a quick reference to some of the most commonly-used SQL commands.

How to Use This Guide:

This guide is in cheat sheet format with self-contained command-line snippets
Jump to any section that is relevant to the task you are trying to complete
When you see highlighted text in this guide's commands, keep in mind that this text should refer to the columns, tables, and data in your own database.
Throughout this guide, the example data values given are all wrapped in apostrophes ('). In SQL, it is necessary to wrap any data values that consist of strings in apostrophes. This isn't required for numeric data, but it also won't cause any issues if you do include apostrophes.

Please note that, while SQL is recognized as a standard, most SQL database programs have their own proprietary extensions. This guide uses MySQL as the example relational database management system (RDBMS), but the commands given will work with other relational database programs, including PostgreSQL, MariaDB, and SQLite. Where there are significant differences between RDBMSs, we have included the alternative commands.

Opening up the Database Prompt (using Socket/Trust Authentication)

By default on Ubuntu 18.04, the root MySQL user can authenticate without a password using the following command:

sudo mysql

To open up a PostgreSQL prompt, use the following command. This example will log you in as the postgres user, which is the included superuser role, but you can replace that with any already-created role:

sudo -u postgres psql

Opening up the Database Prompt (using Password Authentication)

If your root MySQL user is set to authenticate with a password, you can do so with the following command:

mysql -u root -p

If you've already set up a non-root user account for your database, you can also use this method to log in as that user:

mysql -u user -p

The above command will prompt you for your password after you run it. If you'd like to supply your password as part of the command, immediately follow the -p option with your password, with no space between them:

mysql -u root -ppassword

Creating a Database

The following command creates a database with default settings.

CREATE DATABASE database_name;

If you want your database to use a character set and collation different than the defaults, you can specify those using this syntax:

CREATE DATABASE database_name CHARACTER SET character_set COLLATE collation;

Listing Databases

To see what databases exist in your MySQL or MariaDB installation, run the following command:

SHOW DATABASES;

In PostgreSQL, you can see what databases have been created with the following command:

\list

Deleting a Database

To delete a database, including any tables and data held within it, run a command that follows this structure:

DROP DATABASE IF EXISTS database;

Creating a User

To create a user profile for your database without specifying any privileges for it, run the following command:

CREATE USER username IDENTIFIED BY 'password';

PostgreSQL uses a similar, but slightly different, syntax:

CREATE USER user WITH PASSWORD 'password';

If you want to create a new user and grant them privileges in one command, you can do so by issuing a GRANT statement. The following command creates a new user and grants them full privileges to every database and table in the RDBMS:

GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' IDENTIFIED BY 'password';

Deleting a User

Use the following syntax to delete a database user profile:

DROP USER IF EXISTS username;

Note that this command will not by default delete any tables created by the deleted user, and attempts to access such tables may result in errors.

Selecting a Database

Before you can create a table, you first have to tell the RDBMS the database in which you'd like to create it. In MySQL and MariaDB, do so with the following syntax:

USE database;

In PostgreSQL, you must use the following command to select your desired database:

\connect database

Creating a Table

The following command structure creates a new table with the name table, and includes two columns, each with their own specific data type:

CREATE TABLE table ( column_1 column_1_data_type, column_2 column_2_data_taype );

Deleting a Table

To delete a table entirely, including all its data, run the following:

DROP TABLE IF EXISTS table

Inserting Data into a Table

Use the following syntax to populate a table with one row of data:

INSERT INTO table ( column_A, column_B, column_C ) VALUES ( 'data_A', 'data_B', 'data_C' );

You can also populate a table with multiple rows of data using a single command, like this:

INSERT INTO table ( column_A, column_B, column_C ) VALUES ( 'data_1A', 'data_1B', 'data_1C' ),  ( 'data_2A', 'data_2B', 'data_2C' ), ( 'data_3A', 'data_3B', 'data_3C' );

Deleting Data from a Table

To delete a row of data from a table, use the following command structure. Note that value should be the value held in the specified column in the row that you want to delete:

DELETE FROM table WHERE column='value';

Note: If you do not include a WHERE clause in a DELETE statement, as in the following example, it will delete all the data held in a table, but not the columns or the table itself:

DELETE FROM table;

Changing Data in a Table

Use the following syntax to update the data held in a given row. Note that the WHERE clause at the end of the command tells SQL which row to update. value is the value held in column_A that aligns with the row you want to change.

Note: If you fail to include a WHERE clause in an UPDATE statement, the command will replace the data held in every row of the table.

UPDATE table SET column_1 = value_1, column_2 = value_2 WHERE column_A=value;

Inserting a Column

The following command syntax will add a new column to a table:

ALTER TABLE table ADD COLUMN column data_type;

Deleting a Column

A command following this structure will delete a column from a table:

ALTER TABLE table DROP COLUMN column;

Performing Basic Queries

To view all the data from a single column in a table, use the following syntax:

SELECT column FROM table;

To query multiple columns from the same table, separate the column names with a comma:

SELECT column_1, column_2 FROM table;

You can also query every column in a table by replacing the names of the columns with an asterisk (*). In SQL, asterisks act as placeholders to represent “all”:

SELECT * FROM table;

Using WHERE Clauses

You can narrow down the results of a query by appending the SELECT statement with a WHERE clause, like this:

SELECT column FROM table WHERE conditions_that_apply;

For example, you can query all the data from a single row with a syntax like the following. Note that value should be a value held in both the specified column and the row you want to query:

SELECT * FROM table WHERE column = value;

Working with Comparison Operators

A comparison operator in a WHERE clause defines how the specified column should be compared against the value. Here are some common SQL comparison operators:

Operator	What it does
`=`	tests for equality
`!=`	tests for inequality
`<`	tests for less-than
`>`	tests for greater-than
`<=`	tests for less-than or equal-to
`>=`	tests for greater-than or equal-to
`BETWEEN`	tests whether a value lies within a given range
`IN`	tests whether a row's value is contained in a set of specified values
`EXISTS`	tests whether rows exist, given the specified conditions
`LIKE`	tests whether a value matches a specified string
`IS NULL`	tests for `NULL` values
`IS NOT NULL`	tests for all values other than `NULL`

Working with Wildcards

SQL allows the use of wildcard characters. These are useful if you're trying to find a specific entry in a table, but aren't sure of what that entry is exactly.

Asterisks (*) are placeholders that represent “all,” this will query every column in a table:

SELECT * FROM table;

Percentage signs (%) represent zero or more unknown characters.

SELECT * FROM table WHERE column LIKE val%;

Underscores (_) are used to represent a single unknown character:

SELECT * FROM table WHERE column LIKE v_lue;

Counting Entries in a Column

The COUNT function is used to find the number of entries in a given column. The following syntax will return the total number of values held in column:

SELECT COUNT(column) FROM table;

You can narrow down the results of a COUNT function by appending a WHERE clause, like this:

SELECT COUNT(column) FROM table WHERE column=value;

Finding the Average Value in a Column

The AVG function is used to find the average (in this case, the mean) amongst values held in a specific column. Note that the AVG function will only work with columns holding numeric values; when used on a column holding string values, it may return an error or 0:

SELECT AVG(column) FROM table;

Finding the Sum of Values in a Column

The SUM function is used to find the sum total of all the numeric values held in a column:

SELECT SUM(column) FROM table;

As with the AVG function, if you run the SUM function on a column holding string values it may return an error or just 0, depending on your RDBMS.

Finding the Largest Value in a Column

To find the largest numeric value in a column or the last value alphabetically, use the MAX function:

SELECT MAX(column) FROM table;

Finding the Smallest Value in a Column

To find the smallest numeric value in a column or the first value alphabetically, use the MIN function:

SELECT MIN(column) FROM table;

Sorting Results with ORDER BY Clauses

An ORDER BY clause is used to sort query results. The following query syntax returns the values from column_1 and column_2 and sorts the results by the values held in column_1 in ascending order or, for string values, in alphabetical order:

SELECT column_1, column_2 FROM table ORDER BY column_1;

To perform the same action, but order the results in descending or reverse alphabetical order, append the query with DESC:

SELECT column_1, column_2 FROM table ORDER BY column_1 DESC;

Sorting Results with GROUP BY Clauses

The GROUP BY clause is similar to the ORDER BY clause, but it is used to sort the results of a query that includes an aggregate function such as COUNT, MAX, MIN, or SUM. On their own, the aggregate functions described in the previous section will only return a single value. However, you can view the results of an aggregate function performed on every matching value in a column by including a GROUP BY clause.

The following syntax will count the number of matching values in column_2 and group them in ascending or alphabetical order:

SELECT COUNT(column_1), column_2 FROM table GROUP BY column_2;

To perform the same action, but group the results in descending or reverse alphabetical order, append the query with DESC:

SELECT COUNT(column_1), column_2 FROM table GROUP BY column_2 DESC;

Querying Multiple Tables with JOIN Clauses

JOIN clauses are used to create result-sets that combine rows from two or more tables. A JOIN clause will only work if the two tables each have a column with an identical name and data type, as in this example:

SELECT table_1.column_1, table_2.column_2 FROM table_1 JOIN table_2 ON table_1.common_column=table_2.common_column;

This is an example of an INNER JOIN clause. An INNER JOIN will return all the records that have matching values in both tables, but won't show any records that don't have matching values.

It's possible to return all the records from one of two tables, including values that do not have a corresponding match in the other table, by using an outer JOIN clause. Outer JOIN clauses are written as either LEFT JOIN or RIGHT JOIN.

A LEFT JOIN clause returns all the records from the “left” table and only the matching records from the “right” table. In the context of outer JOIN clauses, the left table is the one referenced in the FROM clause, and the right table is any other table referenced after the JOIN statement. The following will show every record from table_1 and only the matching values from table_2. Any values that do not have a match in table_2 will appear as NULL in the result-set:

SELECT table_1.column_1, table_2.column_2 FROM table_1 LEFT JOIN table_2 ON table_1.common_column=table_2.common_column;

A RIGHT JOIN clause functions the same as a LEFT JOIN, but it prints the all the results from the right table, and only the matching values from the left:

SELECT table_1.column_1, table_2.column_2 FROM table_1 RIGHT JOIN table_2 ON table_1.common_column=table_2.common_column;

Combining Multiple SELECT Statements with UNION Clauses

A UNION operator is useful for combining the results of two (or more) SELECT statements into a single result-set:

SELECT column_1 FROM table UNION SELECT column_2 FROM table;

Additionally, the UNION clause can combine two (or more) SELECT statements querying different tables into the same result-set:

SELECT column FROM table_1 UNION SELECT column FROM table_2;

Conclusion

This guide covers some of the more common commands in SQL used to manage databases, users, and tables, and query the contents held in those tables. There are, however, many combinations of clauses and operators that all produce unique result-sets. If you're looking for a more comprehensive guide to working with SQL, we encourage you to check out Oracle's Database SQL Reference.

Additionally, if there are common SQL commands you'd like to see in this guide, please ask or make suggestions in the comments below.

Webinar Series: Building Blocks for Doing CI/CD with Kubernetes

2018-09-12T21:47:10Z

Webinar Series

This article supplements a webinar series on doing CI/CD with Kubernetes. The series discusses how to take a Cloud Native approach to building, testing, and deploying applications, covering release management, Cloud Native tools, Service Meshes, and CI/CD tools that can be used with Kubernetes. It is designed to help developers and businesses that are interested in integrating CI/CD best practices with Kubernetes into their workflows.

This tutorial includes the concepts and commands from the first session of the series, Building Blocks for Doing CI/CD with Kubernetes.

Introduction

If you are getting started with containers, you will likely want to know how to automate building, testing, and deployment. By taking a Cloud Native approach to these processes, you can leverage the right infrastructure APIs to package and deploy applications in an automated way.

Two building blocks for doing automation include container images and container orchestrators. Over the last year or so, Kubernetes has become the default choice for container orchestration. In this first article of the CI/CD with Kubernetes series, you will:

Build container images with Docker, Buildah, and Kaniko.
Set up a Kubernetes cluster with Terraform, and create Deployments and Services.
Extend the functionality of a Kubernetes cluster with Custom Resources.

By the end of this tutorial, you will have container images built with Docker, Buildah, and Kaniko, and a Kubernetes cluster with Deployments, Services, and Custom Resources.

Future articles in the series will cover related topics: package management for Kubernetes, CI/CD tools like Jenkins X and Spinnaker, Services Meshes, and GitOps.

Prerequisites

An Ubuntu 16.04 server with a non-root user account. Follow our Initial Server Setup with Ubuntu 16.04 tutorial for guidance.
Docker installed on your server. Please follow Steps 1 and 2 of How To Install and Use Docker on Ubuntu 16.04 for installation instructions.
A Docker Hub account. For an overview on getting started with Docker Hub, please see these instructions.
A DigitalOcean account and personal access token. Please refer to these instructions to get your access token.
Familiarity with containers and Docker. Please refer to the Webinar Series: Getting Started with Containers for more details.
Familiarity with Kubernetes concepts. Please refer to An Introduction to Kubernetes for more details.

Step 1 — Building Container Images with Docker and Buildah

A container image is a self-contained entity with its own application code, runtime, and dependencies that you can use to create and run containers. You can use different tools to create container images, and in this step you will build containers with two of them: Docker and Buildah.

Building Container Images with Dockerfiles

Docker builds your container images automatically by reading instructions from a Dockerfile, a text file that includes the commands required to assemble a container image. Using the docker image build command, you can create an automated build that will execute the command-line instructions provided in the Dockerfile. When building the image, you will also pass the build context with the Dockerfile, which contains the set of files required to create an environment and run an application in the container image.

Typically, you will create a project folder for your Dockerfile and build context. Create a folder called demo to begin:

mkdir demo
cd demo

Next, create a Dockerfile inside the demo folder:

nano Dockerfile

Add the following content to the file:

~/demo/Dockerfile

FROM ubuntu:16.04

LABEL MAINTAINER neependra@cloudyuga.guru

RUN apt-get update \
    && apt-get install -y nginx \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
    && echo "daemon off;" >> /etc/nginx/nginx.conf

EXPOSE 80
CMD ["nginx"]

This Dockerfile consists of a set of instructions that will build an image to run Nginx. During the build process ubuntu:16.04 will function as the base image, and the nginx package will be installed. Using the CMD instruction, you've also configured nginx to be the default command when the container starts.

Next, you'll build the container image with the docker image build command, using the current directory (.) as the build context. Passing the -t option to this command names the image nkhare/nginx:latest:

sudo docker image build -t nkhare/nginx:latest .

You will see the following output:

OutputSending build context to Docker daemon  49.25MB
Step 1/5 : FROM ubuntu:16.04
 ---> 7aa3602ab41e
Step 2/5 : MAINTAINER neependra@cloudyuga.guru
 ---> Using cache
 ---> 552b90c2ff8d
Step 3/5 : RUN apt-get update     && apt-get install -y nginx     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*     && echo "daemon off;" >> /etc/nginx/nginx.conf
 ---> Using cache
 ---> 6bea966278d8
Step 4/5 : EXPOSE 80
 ---> Using cache
 ---> 8f1c4281309e
Step 5/5 : CMD ["nginx"]
 ---> Using cache
 ---> f545da818f47
Successfully built f545da818f47
Successfully tagged nginx:latest

Your image is now built. You can list your Docker images using the following command:

docker image ls

OutputREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nkhare/nginx        latest              4073540cbcec        3 seconds ago       171MB
ubuntu              16.04               7aa3602ab41e        11 days ago

You can now use the nkhare/nginx:latest image to create containers.

Building Container Images with Project Atomic-Buildah

Buildah is a CLI tool, developed by Project Atomic, for quickly building Open Container Initiative (OCI)-compliant images. OCI provides specifications for container runtimes and images in an effort to standardize industry best practices.

Buildah can create an image either from a working container or from a Dockerfile. It can build images completely in user space without the Docker daemon, and can perform image operations like build, list, push, and tag. In this step, you'll compile Buildah from source and then use it to create a container image.

To install Buildah you will need the required dependencies, including tools that will enable you to manage packages and package security, among other things. Run the following commands to install these packages:

 cd
 sudo apt-get install software-properties-common
 sudo add-apt-repository ppa:alexlarsson/flatpak
 sudo add-apt-repository ppa:gophers/archive
 sudo apt-add-repository ppa:projectatomic/ppa
 sudo apt-get update
 sudo apt-get install bats btrfs-tools git libapparmor-dev libdevmapper-dev libglib2.0-dev libgpgme11-dev libostree-dev libseccomp-dev libselinux1-dev skopeo-containers go-md2man

Because you will compile the buildah source code to create its package, you'll also need to install Go:

sudo apt-get update
sudo curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
sudo tar -xvf go1.8.linux-amd64.tar.gz
sudo mv go /usr/local
sudo echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile
source ~/.profile
go version

You will see the following output, indicating a successful installation:

Output
go version go1.8 linux/amd64

You can now get the buildah source code to create its package, along with the runc binary. runc is the implementation of the OCI container runtime, which you will use to run your Buildah containers.

Run the following commands to install runc and buildah:

mkdir ~/buildah
cd ~/buildah
export GOPATH=`pwd`
git clone https://github.com/projectatomic/buildah ./src/github.com/projectatomic/buildah
cd ./src/github.com/projectatomic/buildah
make runc all TAGS="apparmor seccomp"
sudo cp ~/buildah/src/github.com/opencontainers/runc/runc /usr/bin/.
sudo apt install buildah

Next, create the /etc/containers/registries.conf file to configure your container registries:

sudo nano /etc/containers/registries.conf

Add the following content to the file to specify your registries:

/etc/containers/registries.conf


# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.

[registries.search]
registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com', 'registry.centos.org']

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = []

# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []

The registries.conf configuration file specifies which registries should be consulted when completing image names that do not include a registry or domain portion.

Now run the following command to build an image, using the https://github.com/do-community/rsvpapp repository as the build context. This repository also contains the relevant Dockerfile:

sudo buildah build-using-dockerfile -t rsvpapp:buildah github.com/do-community/rsvpapp

This command creates an image named rsvpapp:buildah from the Dockerfille available in the https://github.com/do-community/rsvpapp repository.

To list the images, use the following command:

sudo buildah images

You will see the following output:

OutputIMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
b0c552b8cf64         docker.io/teamcloudyuga/python:alpine                    Sep 30, 2016 04:39     95.3 MB
22121fd251df         localhost/rsvpapp:buildah                                Sep 11, 2018 14:34     114 MB

One of these images is localhost/rsvpapp:buildah, which you just created. The other, docker.io/teamcloudyuga/python:alpine, is the base image from the Dockerfile.

Once you have built the image, you can push it to Docker Hub. This will allow you to store it for future use. You will first need to login to your Docker Hub account from the command line:

docker login -u your-dockerhub-username -p your-dockerhub-password

Once the login is successful, you will get a file, ~/.docker/config.json, that will contain your Docker Hub credentials. You can then use that file with buildah to push images to Docker Hub.

For example, if you wanted to push the image you just created, you could run the following command, citing the authfile and the image to push:

sudo buildah push --authfile ~/.docker/config.json rsvpapp:buildah docker://your-dockerhub-username/rsvpapp:buildah

You can also push the resulting image to the local Docker daemon using the following command:

sudo buildah push rsvpapp:buildah docker-daemon:rsvpapp:buildah

Finally, take a look at the Docker images you have created:

sudo docker image ls

OutputREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rsvpapp             buildah             22121fd251df        4 minutes ago       108MB
nkhare/nginx        latest              01f0982d91b8        17 minutes ago      172MB
ubuntu              16.04               b9e15a5d1e1a        5 days ago          115MB

As expected, you should now see a new image, rsvpapp:buildah, that has been exported using buildah.

You now have experience building container images with two different tools, Docker and Buildah. Let's move on to discussing how to set up a cluster of containers with Kubernetes.

Step 2 — Setting Up a Kubernetes Cluster on DigitalOcean using kubeadm and Terraform

There are different ways to set up Kubernetes on DigitalOcean. To learn more about how to set up Kubernetes with kubeadm, for example, you can look at How To Create a Kubernetes Cluster Using Kubeadm on Ubuntu 18.04.

Since this tutorial series discusses taking a Cloud Native approach to application development, we'll apply this methodology when setting up our cluster. Specifically, we will automate our cluster creation using kubeadm and Terraform, a tool that simplifies creating and changing infrastructure.

Using your personal access token, you will connect to DigitalOcean with Terraform to provision 3 servers. You will run the kubeadm commands inside of these VMs to create a 3-node Kubernetes cluster containing one master node and two workers.

On your Ubuntu server, create a pair of SSH keys, which will allow password-less logins to your VMs:

ssh-keygen -t rsa

You will see the following output:

OutputGenerating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa):

Press ENTER to save the key pair in the ~/.ssh directory in your home directory, or enter another destination.

Next, you will see the following prompt:

OutputEnter passphrase (empty for no passphrase):

In this case, press ENTER without a password to enable password-less logins to your nodes.

You will see a confirmation that your key pair has been created:

OutputYour identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:lCVaexVBIwHo++NlIxccMW5b6QAJa+ZEr9ogAElUFyY root@3b9a273f18b5
The key's randomart image is:
+---[RSA 2048]----+
|++.E ++o=o*o*o   |
|o   +..=.B = o   |
|.    .* = * o    |
| .   =.o + *     |
|  . . o.S + .    |
|   . +.    .     |
|    . ... =      |
|        o= .     |
|       ...       |
+----[SHA256]-----+

Get your public key by running the following command, which will display it in your terminal:

cat ~/.ssh/id_rsa.pub

Add this key to your DigitalOcean account by following these directions.

Next, install Terraform:

sudo apt-get update
sudo apt-get install unzip
wget https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
unzip terraform_0.11.7_linux_amd64.zip
sudo mv terraform /usr/bin/.
terraform version

You will see output confirming your Terraform installation:

Output
Terraform v0.11.7

Next, run the following commands to install kubectl, a CLI tool that will communicate with your Kubernetes cluster, and to create a ~/.kube directory in your user's home directory:

sudo apt-get install apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/kubernetes.list 
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install kubectl
mkdir -p ~/.kube

Creating the ~/.kube directory will enable you to copy the configuration file to this location. You’ll do that once you run the Kubernetes setup script later in this section. By default, the kubectl CLI looks for the configuration file in the ~/.kube directory to access the cluster.

Next, clone the sample project repository for this tutorial, which contains the Terraform scripts for setting up the infrastructure:

git clone https://github.com/do-community/k8s-cicd-webinars.git

Go to the Terrafrom script directory:

cd k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/

Get a fingerprint of your SSH public key:

ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub | awk '{print $2}'

You will see output like the following, with the highlighted portion representing your key:

Output
MD5:dd:d1:b7:0f:6d:30:c0:be:ed:ae:c7:b9:b8:4a:df:5e

Keep in mind that your key will differ from what's shown here.

Save the fingerprint to an environmental variable so Terraform can use it:

export FINGERPRINT=dd:d1:b7:0f:6d:30:c0:be:ed:ae:c7:b9:b8:4a:df:5e

Next, export your DO personal access token:

export TOKEN=your-do-access-token

Now take a look at the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ project directory:

ls

Outputcluster.tf  destroy.sh  files outputs.tf  provider.tf  script.sh

This folder contains the necessary scripts and configuration files for deploying your Kubernetes cluster with Terraform.

Execute the script.sh script to trigger the Kubernetes cluster setup:

./script.sh

When the script execution is complete, kubectl will be configured to use the Kubernetes cluster you've created.

List the cluster nodes using kubectl get nodes:

kubectl get nodes

OutputNAME                STATUS    ROLES     AGE       VERSION
k8s-master-node     Ready     master    2m        v1.10.0
k8s-worker-node-1   Ready     <none>    1m        v1.10.0
k8s-worker-node-2   Ready     <none>    57s       v1.10.0

You now have one master and two worker nodes in the Ready state.

With a Kubernetes cluster set up, you can now explore another option for building container images: Kaniko from Google.

Step 3 — Building Container Images with Kaniko

Earlier in this tutorial, you built container images with Dockerfiles and Buildah. But what if you could build container images directly on Kubernetes? There are ways to run the docker image build command inside of Kubernetes, but this isn't native Kubernetes tooling. You would have to depend on the Docker daemon to build images, and it would need to run on one of the Pods in the cluster.

A tool called Kaniko allows you to build container images with a Dockerfile on an existing Kubernetes cluster. In this step, you will build a container image with a Dockerfile using Kaniko. You will then push this image to Docker Hub.

In order to push your image to Docker Hub, you will need to pass your Docker Hub credentials to Kaniko. In the previous step, you logged into Docker Hub and created a ~/.docker/config.json file with your login credentials. Let's use this configuration file to create a Kubernetes ConfigMap object to store the credentials inside the Kubernetes cluster. The ConfigMap object is used to store configuration parameters, decoupling them from your application.

To create a ConfigMap called docker-config using the ~/.docker/config.json file, run the following command:

sudo kubectl create configmap docker-config --from-file=$HOME/.docker/config.json

Next, you can create a Pod definition file called pod-kaniko.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory (though it can go anywhere).

First, make sure that you are in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory:

cd ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/

Create the pod-kaniko.yml file:

nano pod-kaniko.yml

Add the following content to the file to specify what will happen when you deploy your Pod. Be sure to replace your-dockerhub-username in the Pod's args field with your own Docker Hub username:

~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/pod-kaniko.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=./Dockerfile",
            "--context=/tmp/rsvpapp/",
            "--destination=docker.io/your-dockerhub-username/rsvpapp:kaniko",
            "--force" ]
    volumeMounts:
      - name: docker-config
        mountPath: /root/.docker/
      - name: demo
        mountPath: /tmp/rsvpapp
  restartPolicy: Never
  initContainers:
    - image: python
      name: demo
      command: ["/bin/sh"]
      args: ["-c", "git clone https://github.com/do-community/rsvpapp.git /tmp/rsvpapp"] 
      volumeMounts:
      - name: demo
        mountPath: /tmp/rsvpapp
  restartPolicy: Never
  volumes:
    - name: docker-config
      configMap:
        name: docker-config
    - name: demo
      emptyDir: {}

This configuration file describes what will happen when your Pod is deployed. First, the Init container will clone the Git repository with the Dockerfile, https://github.com/do-community/rsvpapp.git, into a shared volume called demo. Init containers run before application containers and can be used to run utilties or other tasks that are not desirable to run from your application containers. Your application container, kaniko, will then build the image using the Dockerfile and push the resulting image to Docker Hub, using the credentials you passed to the ConfigMap volume docker-config.

To deploy the kaniko pod, run the following command:

kubectl apply -f pod-kaniko.yml

You will see the following confirmation:

Outputpod/kaniko created

Get the list of pods:

kubectl get pods

You will see the following list:

OutputNAME      READY     STATUS     RESTARTS   AGE
kaniko    0/1       Init:0/1   0          47s

Wait a few seconds, and then run kubectl get pods again for a status update:

kubectl get pods

You will see the following:

OutputNAME      READY     STATUS    RESTARTS   AGE
kaniko    1/1       Running   0          1m

Finally, run kubectl get pods once more for a final status update:

kubectl get pods

OutputNAME      READY     STATUS      RESTARTS   AGE
kaniko    0/1       Completed   0          2m

This sequence of output tells you that the Init container ran, cloning the GitHub repository inside of the demo volume. After that, the Kaniko build process ran and eventually finished.

Check the logs of the pod:

kubectl logs kaniko

You will see the following output:

Output
time="2018-08-02T05:01:24Z" level=info msg="appending to multi args docker.io/your-dockerhub-username/rsvpapp:kaniko"
time="2018-08-02T05:01:24Z" level=info msg="Downloading base image nkhare/python:alpine"
.
.
.
ime="2018-08-02T05:01:46Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-08-02T05:01:48Z" level=info msg="cmd: CMD"
time="2018-08-02T05:01:48Z" level=info msg="Replacing CMD in config with [/bin/sh -c python rsvp.py]"
time="2018-08-02T05:01:48Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-08-02T05:01:49Z" level=info msg="No files were changed, appending empty layer to config."
2018/08/02 05:01:51 mounted blob: sha256:bc4d09b6c77b25d6d3891095ef3b0f87fbe90621bff2a333f9b7f242299e0cfd
2018/08/02 05:01:51 mounted blob: sha256:809f49334738c14d17682456fd3629207124c4fad3c28f04618cc154d22e845b
2018/08/02 05:01:51 mounted blob: sha256:c0cb142e43453ebb1f82b905aa472e6e66017efd43872135bc5372e4fac04031
2018/08/02 05:01:51 mounted blob: sha256:606abda6711f8f4b91bbb139f8f0da67866c33378a6dcac958b2ddc54f0befd2
2018/08/02 05:01:52 pushed blob sha256:16d1686835faa5f81d67c0e87eb76eab316e1e9cd85167b292b9fa9434ad56bf
2018/08/02 05:01:53 pushed blob sha256:358d117a9400cee075514a286575d7d6ed86d118621e8b446cbb39cc5a07303b
2018/08/02 05:01:55 pushed blob sha256:5d171e492a9b691a49820bebfc25b29e53f5972ff7f14637975de9b385145e04
2018/08/02 05:01:56 index.docker.io/your-dockerhub-username/rsvpapp:kaniko: digest: sha256:831b214cdb7f8231e55afbba40914402b6c915ef4a0a2b6cbfe9efb223522988 size: 1243

From the logs, you can see that the kaniko container built the image from the Dockerfile and pushed it to your Docker Hub account.

You can now pull the Docker image. Be sure again to replace your-dockerhub-username with your Docker Hub username:

docker pull your-dockerhub-username/rsvpapp:kaniko

You will see a confirmation of the pull:

Output
kaniko: Pulling from your-dockerhub-username/rsvpapp
c0cb142e4345: Pull complete 
bc4d09b6c77b: Pull complete 
606abda6711f: Pull complete 
809f49334738: Pull complete 
358d117a9400: Pull complete 
5d171e492a9b: Pull complete 
Digest: sha256:831b214cdb7f8231e55afbba40914402b6c915ef4a0a2b6cbfe9efb223522988
Status: Downloaded newer image for your-dockerhub-username/rsvpapp:kaniko

You have now successfully built a Kubernetes cluster and created new images from within the cluster. Let's move on to discussing Deployments and Services.

Step 4 — Create Kubernetes Deployments and Services

Kubernetes Deployments allow you to run your applications. Deployments specify the desired state for your Pods, ensuring consistency across your rollouts. In this step, you will create an Nginx deployment file called deployment.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory to create an Nginx Deployment.

First, open the file:

nano deployment.yml

Add the following configuration to the file to define your Nginx Deployment:

~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

This file defines a Deployment named nginx-deployment that creates three pods, each running an nginx container on port 80.

To deploy the Deployment, run the following command:

kubectl apply -f deployment.yml

You will see a confirmation that the Deployment was created:

Outputdeployment.apps/nginx-deployment created

List your Deployments:

kubectl get deployments

OutputNAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           29s

You can see that the nginx-deployment Deployment has been created and the desired and current count of the Pods are same: 3.

To list the Pods that the Deployment created, run the following command:

kubectl get pods

OutputNAME                                READY     STATUS      RESTARTS   AGE
kaniko                              0/1       Completed   0          9m
nginx-deployment-75675f5897-nhwsp   1/1       Running     0          1m
nginx-deployment-75675f5897-pxpl9   1/1       Running     0          1m
nginx-deployment-75675f5897-xvf4f   1/1       Running     0          1m

You can see from this output that the desired number of Pods are running.

To expose an application deployment internally and externally, you will need to create a Kubernetes object called a Service. Each Service specifies a ServiceType, which defines how the service is exposed. In this example, we will use a NodePort ServiceType, which exposes the Service on a static port on each node.

To do this, create a file, service.yml, in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/ directory:

nano service.yml

Add the following content to define your Service:

~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/service.yml

kind: Service
apiVersion: v1
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30111

These settings define the Service, nginx-service, and specify that it will target port 80 on your Pod. nodePort defines the port where the application will accept external traffic.

To deploy the Service run the following command:

kubectl apply -f service.yml

You will see a confirmation:

Outputservice/nginx-service created

List the Services:

kubectl get service

You will see the following list:

OutputNAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP        5h
nginx-service   NodePort    10.100.98.213   <none>        80:30111/TCP   7s

Your Service, nginx-service, is exposed on port 30111 and you can now access it on any of the node’s public IPs. For example, navigating to http://node_1_ip:30111 or http://node_2_ip:30111 should take you to Nginx's standard welcome page.

Once you have tested the Deployment, you can clean up both the Deployment and Service:

kubectl delete deployment nginx-deployment
kubectl delete service nginx-service

These commands will delete the Deployment and Service you have created.

Now that you have worked with Deployments and Services, let's move on to creating Custom Resources.

Step 5 — Creating Custom Resources in Kubernetes

Kubernetes offers limited but production-ready functionalities and features. It is possible to extend Kubernetes' offerings, however, using its Custom Resources feature. In Kubernetes, a resource is an endpoint in the Kubernetes API that stores a collection of API objects. A Pod resource contains a collection of Pod objects, for instance. With Custom Resources, you can add custom offerings for networking, storage, and more. These additions can be created or removed at any point.

In addition to creating custom objects, you can also employ sub-controllers of the Kubernetes Controller component in the control plane to make sure that the current state of your objects is equal to the desired state. The Kubernetes Controller has sub-controllers for specified objects. For example, ReplicaSet is a sub-controller that makes sure the desired Pod count remains consistent. When you combine a Custom Resource with a Controller, you get a true declarative API that allows you to specify the desired state of your resources.

In this step, you will create a Custom Resource and related objects.

To create a Custom Resource, first make a file called crd.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/ directory:

nano crd.yml

Add the following Custom Resource Definition (CRD):

~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/crd.yml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: webinars.digitalocean.com
spec:
  group: digitalocean.com
  version: v1
  scope: Namespaced
  names:
    plural: webinars
    singular: webinar
    kind: Webinar
    shortNames:
    - wb

To deploy the CRD defined in crd.yml, run the following command:

kubectl create -f crd.yml

You will see a confirmation that the resource has been created:

Outputcustomresourcedefinition.apiextensions.k8s.io/webinars.digitalocean.com created

The crd.yml file has created a new RESTful resource path: /apis/digtialocean.com/v1/namespaces/*/webinars. You can now refer to your objects using webinars, webinar, Webinar, and wb, as you listed them in the names section of the CustomResourceDefinition. You can check the RESTful resource with the following command:

kubectl proxy & curl 127.0.0.1:8001/apis/digitalocean.com

Note: If you followed the initial server setup guide in the prerequisites, then you will need to allow traffic to port 8001 in order for this test to work. Enable traffic to this port with the following command:

sudo ufw allow 8001

You will see the following output:

OutputHTTP/1.1 200 OK
Content-Length: 238
Content-Type: application/json
Date: Fri, 03 Aug 2018 06:10:12 GMT

{
    "apiVersion": "v1", 
    "kind": "APIGroup", 
    "name": "digitalocean.com", 
    "preferredVersion": {
        "groupVersion": "digitalocean.com/v1", 
        "version": "v1"
    }, 
    "serverAddressByClientCIDRs": null, 
    "versions": [
        {
            "groupVersion": "digitalocean.com/v1", 
            "version": "v1"
        }
    ]
}

Next, create the object for using new Custom Resources by opening a file called webinar.yml:

nano webinar.yml

Add the following content to create the object:

~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/webinar.yml

apiVersion: "digitalocean.com/v1"
kind: Webinar
metadata:
  name: webinar1
spec:
  name: webinar
  image: nginx

Run the following command to push these changes to the cluster:

kubectl apply -f webinar.yml

You will see the following output:

Outputwebinar.digitalocean.com/webinar1 created

You can now manage your webinar objects using kubectl. For example:

kubectl get webinar

OutputNAME       CREATED AT
webinar1   21s

You now have an object called webinar1. If there had been a Controller, it would have intercepted the object creation and performed any defined operations.

Deleting a Custom Resource Definition

To delete all of the objects for your Custom Resource, use the following command:

kubectl delete webinar --all

You will see:

Outputwebinar.digitalocean.com "webinar1" deleted

Remove the Custom Resource itself:

kubectl delete crd webinars.digitalocean.com

You will see a confirmation that it has been deleted:

Outputcustomresourcedefinition.apiextensions.k8s.io "webinars.digitalocean.com" deleted

After deletion you will not have access to the API endpoint that you tested earlier with the curl command.

This sequence is an introduction to how you can extend Kubernetes functionalities without modifying your Kubernetes code.

Step 6 — Deleting the Kubernetes Cluster

To destroy the Kubernetes cluster itself, you can use the destroy.sh script from the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom folder. Make sure that you are in this directory:

cd ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom

Run the script:

./destroy.sh

By running this script, you'll allow Terraform to communicate with the DigitalOcean API and delete the servers in your cluster.

Conclusion

In this tutorial, you used different tools to create container images. With these images, you can create containers in any environment. You also set up a Kubernetes cluster using Terraform, and created Deployment and Service objects to deploy and expose your application. Additionally, you extended Kubernetes' functionality by defining a Custom Resource.

You now have a solid foundation to build a CI/CD environment on Kubernetes, which we'll explore in future articles.

How To Install and Secure OpenFaaS Using Docker Swarm on Ubuntu 16.04

2018-09-12T15:47:40Z

The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

Introduction

Serverless architecture hides server instances from the developer and usually exposes an API that allows developers to run their applications in the cloud. This approach helps developers deploy applications quickly, as they can leave provisioning and maintaining instances to the appropriate DevOps teams. It also reduces infrastructure costs, since with the appropriate tooling you can scale your instances per demand.

Applications that run on serverless platforms are called serverless functions. A function is containerized, executable code that's used to perform specific operations. Containerizing applications ensures that you can reproduce a consistent environment on many machines, enabling updating and scaling.

OpenFaaS is a free and open-source framework for building and hosting serverless functions. With official support for both Docker Swarm and Kubernetes, it lets you deploy your applications using the powerful API, command-line interface, or Web UI. It comes with built-in metrics provided by Prometheus and supports auto-scaling on demand, as well as scaling from zero.

In this tutorial, you'll set up and use OpenFaaS with Docker Swarm running on Ubuntu 16.04, and secure its Web UI and API by setting up Traefik with Let's Encypt. This ensures secure communication between nodes in the cluster, as well as between OpenFaaS and its operators.

Prerequisites

To follow this tutorial, you'll need:

Ubuntu 16.04 running on your local machine. You can use other distributions and operating systems, but make sure you use the appropriate OpenFaaS scripts for your operating system and install all of the dependencies listed in these prerequisites.
git, curl, and jq installed on your local machine. You'll use git to clone the OpenFaaS repository, curl to test the API, and jq to transform raw JSON responses from the API to human-readable JSON. To install the required dependencies for this setup, use the following commands: sudo apt-get update && sudo apt-get install git curl jq
Docker installed, following Steps 1 and 2 of How To Install and Use Docker on Ubuntu 16.04.
A Docker Hub account. To deploy functions to OpenFaaS, they will need to be published on a public container registry. We'll use Docker Hub for this tutorial, since it's both free and widely used. Be sure to authenticate with Docker on your local machine by using the docker login command.
Docker Machine installed, following How To Provision and Manage Remote Docker Hosts with Docker Machine on Ubuntu 16.04.
A DigitalOcean personal access token. To create a token, follow these instructions.
A Docker Swarm cluster of 3 nodes, provisioned by following How to Create a Cluster of Docker Containers with Docker Swarm and DigitalOcean on Ubuntu 16.04.
A fully registered domain name with an A record pointing to one of the instances in the Docker Swarm. Throughout the tutorial, you'll see example.com as an example domain. You should replace this with your own domain, which you can either purchase on Namecheap, or get for free on Freenom. You can also use a different domain registrar of your choice.

Step 1 — Downloading OpenFaaS and Installing the OpenFaaS CLI

To deploy OpenFaaS to your Docker Swarm, you will need to download the deployment manifests and scripts. The easiest way to obtain them is to clone the official OpenFaas repository and check out the appropriate tag, which represents an OpenFaaS release.

In addition to cloning the repository, you'll also install the FaaS CLI, a powerful command-line utility that you can use to manage and deploy new functions from your terminal. It provides templates for creating your own functions in most major programming languages. In Step 7, you'll use it to create a Python function and deploy it on OpenFaaS.

For this tutorial, you'll deploy OpenFaaS v0.8.9. While the steps for deploying other versions should be similar, make sure to check out the project changelog to ensure there are no breaking changes.

First, navigate to your home directory and run the following command to clone the repository to the ~/faas directory:

cd ~
git clone https://github.com/openfaas/faas.git

Navigate to the newly-created ~/faas directory:

cd ~/faas

When you clone the repository, you'll get files from the master branch that contain the latest changes. Because breaking changes can get into the master branch, it's not recommended for use in production. Instead, let's check out the 0.8.9 tag:

git checkout 0.8.9

The output contains a message about the successful checkout and a warning about committing changes to this branch:

Output
Note: checking out '0.8.9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 8f0d2d1 Expose scale-function endpoint

If you see any errors, make sure to resolve them by following the on-screen instructions before continuing.

With the OpenFaaS repository downloaded, complete with the necessary manifest files, let's proceed to installing the FaaS CLI.

The easiest way to install the FaaS CLI is to use the official script. In your terminal, navigate to your home directory and download the script using the following command:

cd ~
curl -sSL -o faas-cli.sh https://cli.openfaas.com

This will download the faas-cli.sh script to your home directory. Before executing the script, it's a good idea to check the contents:

less faas-cli.sh

You can exit the preview by pressing q. Once you have verified content of the script, you can proceed with the installation by giving executable permissions to the script and executing it. Execute the script as root so it will automatically copy to your PATH:

chmod +x faas-cli.sh
sudo ./faas-cli.sh

The output contains information about the installation progress and the CLI version that you've installed:

Outputx86_64
Downloading package https://github.com/openfaas/faas-cli/releases/download/0.6.17/faas-cli as /tmp/faas-cli
Download complete.

Running as root - Attempting to move faas-cli to /usr/local/bin
New version of faas-cli installed to /usr/local/bin
Creating alias 'faas' for 'faas-cli'.
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|

CLI:
 commit:  b5597294da6dd98457434fafe39054c993a5f7e7
 version: 0.6.17

If you see an error, make sure to resolve it by following the on-screen instructions before continuing with the tutorial.

At this point, you have the FaaS CLI installed. To learn more about commands you can use, execute the CLI without any arguments:

faas-cli

The output shows available commands and flags:

Output  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|


Manage your OpenFaaS functions from the command line

Usage:
  faas-cli [flags]
  faas-cli [command]

Available Commands:
  build          Builds OpenFaaS function containers
  cloud          OpenFaaS Cloud commands
  deploy         Deploy OpenFaaS functions
  help           Help about any command
  invoke         Invoke an OpenFaaS function
  list           List OpenFaaS functions
  login          Log in to OpenFaaS gateway
  logout         Log out from OpenFaaS gateway
  new            Create a new template in the current folder with the name given as name
  push           Push OpenFaaS functions to remote registry (Docker Hub)
  remove         Remove deployed OpenFaaS functions
  store          OpenFaaS store commands
  template       Downloads templates from the specified github repo
  version        Display the clients version information

Flags:
      --filter string   Wildcard to match with function names in YAML file
  -h, --help            help for faas-cli
      --regex string    Regex to match with function names in YAML file
  -f, --yaml string     Path to YAML file describing function(s)

Use "faas-cli [command] --help" for more information about a command.

You have now successfully obtained the OpenFaaS manifests and installed the FaaS CLI, which you can use to manage your OpenFaaS instance from your terminal.

The ~/faas directory contains files from the 0.8.9 release, which means you can now deploy OpenFaaS to your Docker Swarm. Before doing so, let's modify the deployment manifest file to include Traefik, which will secure your OpenFaaS setup by setting up Let's Encrypt.

Step 2 — Configuring Traefik

Traefik is a Docker-aware reverse proxy that comes with SSL support provided by Let's Encrypt. SSL protocol ensures that you communicate with the Swarm cluster securely by encrypting the data you send and receive between nodes.

To use Traefik with OpenFaaS, you need to modify the OpenFaaS deployment manifest to include Traefik and tell OpenFaaS to use Traefik instead of directly exposing its services to the internet.

Navigate back to the ~/faas directory and open the OpenFaaS deployment manifest in a text editor:

cd ~/faas
nano ~/faas/docker-compose.yml

Note: The Docker Compose manifest file uses YAML formatting, which strictly forbids tabs and requires two spaces for indentation. The manifest will fail to deploy if the file is incorrectly formatted.

The OpenFaaS deployment is comprised of several services, defined under the services directive, that provide the dependencies needed to run OpenFaaS, the OpenFaaS API and Web UI, and Prometheus and AlertManager (for handling metrics).

At the beginning of the services section, add a new service called traefik, which uses the traefik:v1.6 image for the deployment:

~/faas/docker-compose.yml

version: "3.3"
services:
    traefik:
        image: traefik:v1.6
    gateway:
         ...

The Traefik image is coming from the Traefik Docker Hub repository, where you can find a list of all available images.

Next, let's instruct Docker to run Traefik using the command directive. This will run Traefik, configure it to work with Docker Swarm, and provide SSL using Let's Encrypt. The following flags will configure Traefik:

--docker.*: These flags tell Traefik to use Docker and specify that it's running in a Docker Swarm cluster.
--web=true: This flag enables Traefik's Web UI.
--defaultEntryPoints and --entryPoints: These flags define entry points and protocols to be used. In our case this includes HTTP on port 80 and HTTPS on port 443.
--acme.*: These flags tell Traefik to use ACME to generate Let's Encrypt certificates to secure your OpenFaaS cluster with SSL.

Make sure to replace the example.com domain placeholders in the --acme.domains and --acme.email flags with the domain you're going to use to access OpenFaaS. You can specify multiple domains by separating them with a comma and space. The email address is for SSL notifications and alerts, including certificate expiry alerts. In this case, Traefik will handle renewing certificates automatically, so you can ignore expiry alerts.

Add the following block of code below the image directive, and above gateway:

~/faas/docker-compose.yml

...
    traefik:
        image: traefik:v1.6
        command: -c --docker=true
            --docker.swarmmode=true
            --docker.domain=traefik
            --docker.watch=true
            --web=true
            --defaultEntryPoints='http,https'
            --entryPoints='Name:https Address::443 TLS'
            --entryPoints='Name:http Address::80'
            --acme=true
            --acme.entrypoint='https'
            --acme.httpchallenge=true
            --acme.httpchallenge.entrypoint='http'
            --acme.domains='example.com, www.example.com'
            --acme.email='sammy@example.com'
            --acme.ondemand=true
            --acme.onhostrule=true
            --acme.storage=/etc/traefik/acme/acme.json
...

With the command directive in place, let's tell Traefik what ports to expose to the internet. Traefik uses port 8080 for its operations, while OpenFaaS will use port 80 for non-secure communication and port 443 for secure communication.

Add the following ports directive below the command directive. The port-internet:port-docker notation ensures that the port on the left side is exposed by Traefik to the internet and maps to the container's port on the right side:

~/faas/docker-compose.yml

        ...
        command:
            ...
        ports:
            - 80:80
            - 8080:8080
            - 443:443
        ...

Next, using the volumes directive, mount the Docker socket file from the host running Docker to Traefik. The Docker socket file communicates with the Docker API in order to manage your containers and get details about them, such as number of containers and their IP addresses. You will also mount the volume called acme, which we'll define later in this step.

The networks directive instructs Traefik to use the functions network, which is deployed along with OpenFaaS. This network ensures that functions can communicate with other parts of the system, including the API.

The deploy directive instructs Docker to run Traefik only on the Docker Swarm manager node.

Add the following directives below the ports directive:

~/faas/docker-compose.yml

        ...
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
            - "acme:/etc/traefik/acme"
        networks:
            - functions
        deploy:
            placement:
                constraints: [node.role == manager]

At this point, the traefik service block should look like this:

~/faas/docker-compose.yml

version: "3.3"
services:
    traefik:
        image: traefik:v1.6
        command: -c --docker=true
            --docker.swarmmode=true
            --docker.domain=traefik
            --docker.watch=true
            --web=true
            --defaultEntryPoints='http,https'
            --entryPoints='Name:https Address::443 TLS'
            --entryPoints='Name:http Address::80'            
            --acme=true
            --acme.entrypoint='https'
            --acme.httpchallenge=true
            --acme.httpchallenge.entrypoint='http'
            --acme.domains='example.com, www.example.com'
            --acme.email='sammy@example.com'
            --acme.ondemand=true
            --acme.onhostrule=true
            --acme.storage=/etc/traefik/acme/acme.json
        ports:
            - 80:80
            - 8080:8080
            - 443:443
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
            - "acme:/etc/traefik/acme"
        networks:
          - functions
        deploy:
          placement:
            constraints: [node.role == manager]

    gateway:
        ...

While this configuration ensures that Traefik will be deployed with OpenFaaS, you also need to configure OpenFaaS to work with Traefik. By default, the gateway service is configured to run on port 8080, which overlaps with Traefik.

The gateway service provides the API gateway you can use to deploy, run, and manage your functions. It handles metrics (via Prometheus) and auto-scaling, and hosts the Web UI.

Our goal is to expose the gateway service using Traefik instead of exposing it directly to the internet.

Locate the gateway service, which should look like this:

~/faas/docker-compose.yml

...
    gateway:
        ports:
            - 8080:8080
        image: openfaas/gateway:0.8.7
        networks:
            - functions
        environment:
            functions_provider_url: "http://faas-swarm:8080/"
            read_timeout:  "300s"        # Maximum time to read HTTP request
            write_timeout: "300s"        # Maximum time to write HTTP response
            upstream_timeout: "300s"     # Maximum duration of upstream function call - should be more than read_timeout and write_timeout
            dnsrr: "true"               # Temporarily use dnsrr in place of VIP while issue persists on PWD
            faas_nats_address: "nats"
            faas_nats_port: 4222
            direct_functions: "true"    # Functions are invoked directly over the overlay network
            direct_functions_suffix: ""
            basic_auth: "${BASIC_AUTH:-true}"
            secret_mount_path: "/run/secrets/"
            scale_from_zero: "false"
        deploy:
            resources:
                # limits:   # Enable if you want to limit memory usage
                #     memory: 200M
                reservations:
                    memory: 100M
            restart_policy:
                condition: on-failure
                delay: 5s
                max_attempts: 20
                window: 380s
            placement:
                constraints:
                    - 'node.platform.os == linux'
        secrets:
            - basic-auth-user
            - basic-auth-password
...

Remove the ports directive from the service to avoid exposing the gateway service directly.

Next, add the following lables directive to the deploy section of the gateway service. This directive exposes the /ui, /system, and /function endpoints on port 8080 over Traefik:

~/faas/docker-compose.yml

        ...
        deploy:
            labels:
                - traefik.port=8080
                - traefik.frontend.rule=PathPrefix:/ui,/system,/function
            resources:
            ...

The /ui endpoint exposes the OpenFaaS Web UI, which is covered in the Step 6 of this tutorial. The /system endpoint is the API endpoint used to manage OpenFaaS, while the /function endpoint exposes the API endpoints for managing and running functions. Step 5 of this tutorial covers the OpenFaaS API in detail.

After modifications, your gateway service should look like this:

~/faas/docker-compose.yml

...
    gateway:       
        image: openfaas/gateway:0.8.7
        networks:
            - functions
        environment:
            functions_provider_url: "http://faas-swarm:8080/"
            read_timeout:  "300s"        # Maximum time to read HTTP request
            write_timeout: "300s"        # Maximum time to write HTTP response
            upstream_timeout: "300s"     # Maximum duration of upstream function call - should be more than read_timeout and write_timeout
            dnsrr: "true"               # Temporarily use dnsrr in place of VIP while issue persists on PWD
            faas_nats_address: "nats"
            faas_nats_port: 4222
            direct_functions: "true"    # Functions are invoked directly over the overlay network
            direct_functions_suffix: ""
            basic_auth: "${BASIC_AUTH:-true}"
            secret_mount_path: "/run/secrets/"
            scale_from_zero: "false"
        deploy:
            labels:
                - traefik.port=8080
                - traefik.frontend.rule=PathPrefix:/ui,/system,/function
            resources:
                # limits:   # Enable if you want to limit memory usage
                #     memory: 200M
                reservations:
                    memory: 100M
            restart_policy:
                condition: on-failure
                delay: 5s
                max_attempts: 20
                window: 380s
            placement:
                constraints:
                    - 'node.platform.os == linux'
        secrets:
            - basic-auth-user
            - basic-auth-password
...

Finally, let's define the acme volume used for storing Let's Encrypt certificates. We can define an empty volume, meaning data will not persist if you destroy the container. If you destroy the container, the certificates will be regenerated the next time you start Traefik.

Add the following volumes directive on the last line of the file:

~/faas/docker-compose.yml

...
volumes:
    acme:

Once you're done, save the file and close your text editor. At this point, you've configured Traefik to protect your OpenFaaS deployment and Docker Swarm. Now you're ready to deploy it along with OpenFaaS on your Swarm cluster.

Step 3 — Deploying OpenFaaS

Now that you have prepared the OpenFaaS deployment manifest, you're ready to deploy it and start using OpenFaaS. To deploy, you'll use the deploy_stack.sh script. This script is meant to be used on Linux and macOS operating systems, but in the OpenFaaS directory you can also find appropriate scripts for Windows and ARM systems.

Before deploying OpenFaaS, you will need to instruct docker-machine to execute Docker commands from the script on one of the machines in the Swarm. For this tutorial, let's use the Swarm manager.

If you have the docker-machine use command configured, you can use it:

docker-machine use node-1

If not, use the following command:

eval $(docker-machine env node-1)

The deploy_stack.sh script deploys all of the resources required for OpenFaaS to work as expected, including configuration files, network settings, services, and credentials for authorization with the OpenFaaS server.

Let's execute the script, which will take several minutes to finish deploying:

~/faas/deploy_stack.sh

The output shows a list of resources that are created in the deployment process, as well as the credentials you will use to access the OpenFaaS server and the FaaS CLI command.

Write down these credentials, as you will need them throughout the tutorial to access the Web UI and the API:

OutputAttempting to create credentials for gateway..
roozmk0y1jkn17372a8v9y63g
q1odtpij3pbqrmmf8msy3ampl
[Credentials]
 username: admin
 password: your_openfaas_password
 echo -n your_openfaas_password | faas-cli login --username=admin --password-stdin

Enabling basic authentication for gateway..

Deploying OpenFaaS core services
Creating network func_functions
Creating config func_alertmanager_config
Creating config func_prometheus_config
Creating config func_prometheus_rules
Creating service func_alertmanager
Creating service func_traefik
Creating service func_gateway
Creating service func_faas-swarm
Creating service func_nats
Creating service func_queue-worker
Creating service func_prometheus

If you see any errors, follow the on-screen instructions to resolve them before continuing the tutorial.

Before continuing, let's authenticate the FaaS CLI with the OpenFaaS server using the command provided by the deployment script.

The script outputted the flags you need to provide to the command, but you will need to add an additional flag, --gateway, with the address of your OpenFaaS server, as the FaaS CLI assumes the gateway server is running on localhost:

echo -n your_openfaas_password | faas-cli login --username=admin --password-stdin --gateway https://example.com

The output contains a message about successful authorization:

OutputCalling the OpenFaaS server to validate the credentials...
credentials saved for admin https://example.com

At this point, you have a fully-functional OpenFaaS server deployed on your Docker Swarm cluster, as well as the FaaS CLI configured to use your newly deployed server. Before testing how to use OpenFaaS, let's deploy some sample functions to get started.

Step 4 — Deploying OpenFaaS Sample Functions

Initially, OpenFaaS comes without any functions deployed. To start testing and using it, you will need some functions.

The OpenFaaS project hosts some sample functions, and you can find a list of available functions along with their deployment manifests in the OpenFaaS repository. Some of the sample functions include nodeinfo, for showing information about the node where a function is running, wordcount, for counting the number of words in a passed request, and markdown, for converting passed markdown input to HTML output.

The stack.yml manifest in the ~/faas directory deploys several sample functions along with the functions mentioned above. You can deploy it using the FaaS CLI.

Run the following faas-cli command, which takes the path to the stack manifest and the address of your OpenFaaS server:

faas-cli deploy -f ~/faas/stack.yml --gateway https://example.com

The output contains status codes and messages indicating whether or not the deployment was successful:

OutputDeploying: wordcount.

Deployed. 200 OK.
URL: https://example.com/function/wordcount

Deploying: base64.

Deployed. 200 OK.
URL: https://example.com/function/base64

Deploying: markdown.

Deployed. 200 OK.
URL: https://example.com/function/markdown

Deploying: hubstats.

Deployed. 200 OK.
URL: https://example.com/function/hubstats

Deploying: nodeinfo.

Deployed. 200 OK.
URL: https://example.com/function/nodeinfo

Deploying: echoit.

Deployed. 200 OK.
URL: https://example.com/function/echoit

If you see any errors, make sure to resolve them by following the on-screen instructions.

Once the stack deployment is done, list all of the functions to make sure they're deployed and ready to be used:

faas-cli list --gateway https://example.com

The output contains a list of functions, along with their replica numbers and an invocations count:

OutputFunction                        Invocations     Replicas
markdown                        0               1
wordcount                       0               1
base64                          0               1
nodeinfo                        0               1
hubstats                        0               1
echoit                          0               1

If you don't see your functions here, make sure the faas-cli deploy command executed successfully.

You can now use the sample OpenFaaS functions to test and demonstrate how to use the API, Web UI, and CLI. In the next step, you'll start by using the OpenFaaS API to list and run functions.

Step 5 — Using the OpenFaaS API

OpenFaaS comes with a powerful API that you can use to manage and execute your serverless functions. Let's use Swagger, a tool for architecting, testing, and documenting APIs, to browse the API documentation, and then use the API to list and run functions.

With Swagger, you can inspect the API documentation to find out what endpoints are available and how you can use them. In the OpenFaaS repository, you can find the Swagger API specification, which can be used with the Swagger editor to convert the specification to human-readable form.

Navigate your web browser to http://editor.swagger.io/. You should be welcomed with the following screen:

Here you'll find a text editor containing the source code for the sample Swagger specification, and the human-readable API documentation on the right.

Let's import the OpenFaaS Swagger specification. In the top menu, click on the File button, and then on Import URL:

You'll see a pop-up, where you need to enter the address of the Swagger API specification. If you don't see the pop-up, make sure pop-ups are enabled for your web browser.

In the field, enter the link to the Swagger OpenFaaS API specification: https://raw.githubusercontent.com/openfaas/faas/master/api-docs/swagger.yml

After clicking on the OK button, the Swagger editor will show you the API reference for OpenFaaS, which should look like this:

On the left side you can see the source of the API reference file, while on the right side you can see a list of endpoints, along with short descriptions. Clicking on an endpoint shows you more details about it, including what parameters it takes, what method it uses, and possible responses:

Once you know what endpoints are available and what parameters they expect, you can use them to manage your functions.

Next, you'll use a curl command to communicate with the API, so navigate back to your terminal. With the -u flag, you will be able to pass the admin:your_openfaas_password pair that you got in Step 3, while the -X flag will define the request method. You will also pass your endpoint URL, https://example.com/system/functions:

curl -u admin:your_openfaas_password -X GET https://example.com/system/functions

You can see the required method for each endpoint in the API docs.

In Step 4, you deployed several sample functions, which should appear in the output:

Output[{"name":"base64","image":"functions/alpine:latest","invocationCount":0,"replicas":1,"envProcess":"base64","availableReplicas":0,"labels":{"com.openfaas.function":"base64","function":"true"}},{"name":"nodeinfo","image":"functions/nodeinfo:latest","invocationCount":0,"replicas":1,"envProcess":"","availableReplicas":0,"labels":{"com.openfaas.function":"nodeinfo","function":"true"}},{"name":"hubstats","image":"functions/hubstats:latest","invocationCount":0,"replicas":1,"envProcess":"","availableReplicas":0,"labels":{"com.openfaas.function":"hubstats","function":"true"}},{"name":"markdown","image":"functions/markdown-render:latest","invocationCount":0,"replicas":1,"envProcess":"","availableReplicas":0,"labels":{"com.openfaas.function":"markdown","function":"true"}},{"name":"echoit","image":"functions/alpine:latest","invocationCount":0,"replicas":1,"envProcess":"cat","availableReplicas":0,"labels":{"com.openfaas.function":"echoit","function":"true"}},{"name":"wordcount","image":"functions/alpine:latest","invocationCount":0,"replicas":1,"envProcess":"wc","availableReplicas":0,"labels":{"com.openfaas.function":"wordcount","function":"true"}}]

If you don't see output that looks like this, or if you see an error, follow the on-screen instructions to resolve the problem before continuing with the tutorial. Make sure you're sending the request to the correct endpoint using the recommended method and the right credentials. You can also check the logs for the gateway service using the following command:

docker service logs func_gateway

By default, the API response to the curl call returns raw JSON without new lines, which is not human-readable. To parse it, pipe curl's response to the jq utility, which will convert the JSON to human-readable form:

curl -u admin:your_openfaas_password -X GET https://example.com/system/functions | jq

The output is now in human-readable form. You can see the function name, which you can use to manage and invoke functions with the API, the number of invocations, as well as information such as labels and number of replicas, relevant to Docker:

Output[
  {
    "name": "base64",
    "image": "functions/alpine:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "base64",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "base64",
      "function": "true"
    }
  },
  {
    "name": "nodeinfo",
    "image": "functions/nodeinfo:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "nodeinfo",
      "function": "true"
    }
  },
  {
    "name": "hubstats",
    "image": "functions/hubstats:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "hubstats",
      "function": "true"
    }
  },
  {
    "name": "markdown",
    "image": "functions/markdown-render:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "markdown",
      "function": "true"
    }
  },
  {
    "name": "echoit",
    "image": "functions/alpine:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "cat",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "echoit",
      "function": "true"
    }
  },
  {
    "name": "wordcount",
    "image": "functions/alpine:latest",
    "invocationCount": 0,
    "replicas": 1,
    "envProcess": "wc",
    "availableReplicas": 0,
    "labels": {
      "com.openfaas.function": "wordcount",
      "function": "true"
    }
  }
]

Let's take one of these functions and execute it, using the API /function/function-name endpoint. This endpoint is available over the POST method, where the -d flag allows you to send data to the function.

For example, let's run the following curl command to execute the echoit function, which comes with OpenFaaS out of the box and outputs the string you've sent it as a request. You can use the string "Sammy The Shark" to demonstrate:

curl -u admin:your_openfaas_password -X POST https://example.com/function/func_echoit -d "Sammy The Shark"

The output will show you Sammy The Shark:

OutputSammy The Shark

If you see an error, follow the on-screen logs to resolve the problem before continuing with the tutorial. You can also check the gateway service's logs.

At this point, you've used the OpenFaaS API to manage and execute your functions. Let's now take a look at the OpenFaaS Web UI.

Step 6 — Using the OpenFaaS Web UI

OpenFaaS comes with a Web UI that you can use to add new and execute installed functions. In this step, you will install a function for generating QR Codes from the FaaS Store and generate a sample code.

To begin, point your web browser to https://example.com/ui/. Note that the trailing slash is required to avoid a "not found" error.

In the HTTP authentication dialogue box, enter the username and password you got when deploying OpenFaaS in Step 3.

Once logged in, you will see available functions on the left side of the screen, along with the Deploy New Functions button used to install new functions.

Click on Deploy New Functions to deploy a new function. You will see the FaaS Store window, which provides community-tested functions that you can install with a single click:

In addition to these functions, you can also deploy functions manually from a Docker image.

For this tutorial, you will deploy the QR Code Generator function from the FaaS Store. Locate the QR Code Generator - Go item in the list, click on it, and then click the Deploy button at the bottom of the window:

After clicking Deploy, the Deploy A New Function window will close and the function will be deployed. In the list at the left side of the window you will see a listing for the qrcode-go function. Click on this entry to select it. The main function window will show the function name, number of replicas, invocation count, and image, along with the option to invoke the function:

Let's generate a QR code containing the URL with your domain. In the Request body field, type the content of the QR code you'd like to generate; in our case, this will be "example.com". Once you're done, click the Invoke button.

When you select either the Text or JSON output option, the function will output the file's content, which is not usable or human-readable:

You can download a response. which in our case will be a PNG file with the QR code. To do this, select the Download option, and then click Invoke once again. Shortly after, you should have the QR code downloaded, which you can open with the image viewer of your choice:

In addition to deploying functions from the FaaS store or from Docker images, you can also create your own functions. In the next step, you will create a Python function using the FaaS command-line interface.

Step 7 — Creating Functions With the FaaS CLI

In the previous steps, you configured the FaaS CLI to work with your OpenFaaS server. The FaaS CLI is a command-line interface that you can use to manage OpenFaaS and install and run functions, just like you would over the API or using the Web UI.

Compared to the Web UI or the API, the FaaS CLI has templates for many programming languages that you can use to create your own functions. It can also build container images based on your function code and push images to an image registry, such as Docker Hub.

In this step, you will create a function, publish it to Docker Hub, and then run it on your OpenFaaS server. This function will be similar to the default echoit function, which returns input passed as a request.

We will use Python to write our function. If you want to learn more about Python, you can check out our How To Code in Python 3 tutorial series and our How To Code in Python eBook.

Before creating the new function, let's create a directory to store FaaS functions and navigate to it:

mkdir ~/faas-functions
cd ~/faas-functions

Execute the following command to create a new Python function called echo-input. Make sure to replace your-docker-hub-username with your Docker Hub username, as you'll push the function to Docker Hub later:

faas-cli new echo-input --lang python --prefix your-docker-hub-username --gateway https://example.com

The output contains confirmation about the successful function creation. If you don't have templates downloaded, the CLI will download templates in your current directory:

Output2018/05/13 12:13:06 No templates found in current directory.
2018/05/13 12:13:06 Attempting to expand templates from https://github.com/openfaas/templates.git
2018/05/13 12:13:11 Fetched 12 template(s) : [csharp dockerfile go go-armhf node node-arm64 node-armhf python python-armhf python3 python3-armhf ruby] from https://github.com/openfaas/templates.git
Folder: echo-input created.
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|


Function created in folder: echo-input
Stack file written: echo-input.yml

The result of the faas-cli new command is a newly-created ~/faas-fucntions/echo-input directory containing the function's code and the echo-input.yml file. This file includes information about your function: what language it's in, its name, and the server you will deploy it on.

Navigate to the ~/faas-fucntions/echo-input directory:

cd ~/faas-fucntions/echo-input

To see content of the directory, execute:

ls

The directory contains two files: handler.py, which contains the code for your function, and requirements.txt, which contains the Python modules required by the function.

Since we don't currently require any non-default Python modules, the requirements.txt file is empty. You can check that by using the cat command:

cat requirements.txt

Next, let's write a function that will return a request as a string.

The handler.py file already has the sample handler code, which returns a received response as a string. Let's take a look at the code:

nano handler.py

The default function is called handle and takes a single parameter, req, that contains a request that's passed to the function when it's invoked. The function does only one thing, returning the passed request back as the response:

def handle(req):
    """handle a request to the function
    Args:
        req (str): request body
    """

    return req

Let's modify it to include additional text, replacing the string in the return directive as follows:

    return "Received message: " + req

Once you're done, save the file and close your text editor.

Next, let's build a Docker image from the function's source code. Navigate to the faas-functions directory where the echo-input.yml file is located:

cd ~/faas-functions

The following command builds the Docker image for your function:

faas-cli build -f echo-input.yml

The output contains information about the build progress:

Output[0] > Building echo-input.
Clearing temporary build folder: ./build/echo-input/
Preparing ./echo-input/ ./build/echo-input/function
Building: sammy/echo-input with python template. Please wait..
Sending build context to Docker daemon  7.168kB
Step 1/16 : FROM python:2.7-alpine
 ---> 5fdd069daf25
Step 2/16 : RUN apk --no-cache add curl     && echo "Pulling watchdog binary from Github."     && curl -sSL https://github.com/openfaas/faas/releases/download/0.8.0/fwatchdog > /usr/bin/fwatchdog     && chmod +x /usr/bin/fwatchdog     && apk del curl --no-cache
 ---> Using cache
 ---> 247d4772623a
Step 3/16 : WORKDIR /root/
 ---> Using cache
 ---> 532cc683d67b
Step 4/16 : COPY index.py           .
 ---> Using cache
 ---> b4b512152257
Step 5/16 : COPY requirements.txt   .
 ---> Using cache
 ---> 3f9cbb311ab4
Step 6/16 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> dd7415c792b1
Step 7/16 : RUN mkdir -p function
 ---> Using cache
 ---> 96c25051cefc
Step 8/16 : RUN touch ./function/__init__.py
 ---> Using cache
 ---> 77a9db274e32
Step 9/16 : WORKDIR /root/function/
 ---> Using cache
 ---> 88a876eca9e3
Step 10/16 : COPY function/requirements.txt .
 ---> Using cache
 ---> f9ba5effdc5a
Step 11/16 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 394a1dd9e4d7
Step 12/16 : WORKDIR /root/
 ---> Using cache
 ---> 5a5893c25b65
Step 13/16 : COPY function           function
 ---> eeddfa67018d
Step 14/16 : ENV fprocess="python index.py"
 ---> Running in 8e53df4583f2
Removing intermediate container 8e53df4583f2
 ---> fb5086bc7f6c
Step 15/16 : HEALTHCHECK --interval=1s CMD [ -e /tmp/.lock ] || exit 1
 ---> Running in b38681a71378
Removing intermediate container b38681a71378
 ---> b04c045b0994
Step 16/16 : CMD ["fwatchdog"]
 ---> Running in c5a11078df3d
Removing intermediate container c5a11078df3d
 ---> bc5f08157c5a
Successfully built bc5f08157c5a
Successfully tagged sammy/echo-input:latest
Image: your-docker-hub-username/echo-input built.
[0] < Building echo-input done.
[0] worker done.

If you get an error, make sure to resolve it by following the on-screen instructions before deploying the function.

You will need to containerize your OpenFaaS function in order to deploy it. Containerizing applications ensures that the environment needed to run your application can be easily reproduced, and your application can be easily deployed, scaled, and updated.

For this tutorial, we'll use Docker Hub, as it's a free solution, but you can use any container registry, including your own private registry.

Run the following command to push the image you built to your specified repository on Docker Hub:

faas-cli push -f echo-input.yml

Pushing will take several minutes, depending on your internet connection speed. The output contains the image's upload progress:

Output[0] > Pushing echo-input.
The push refers to repository [docker.io/sammy/echo-input]
320ea573b385: Pushed 
9d87e56f5d0c: Pushed 
6f79b75e7434: Pushed 
23aac2d8ecf2: Pushed 
2bec17d09b7e: Pushed 
e5a0e5ab3be6: Pushed 
e9c8ca932f1b: Pushed 
beae1d55b4ce: Pushed 
2fcae03ed1f7: Pushed 
62103d5daa03: Mounted from library/python 
f6ac6def937b: Mounted from library/python 
55c108c7613c: Mounted from library/python 
e53f74215d12: Mounted from library/python 
latest: digest: sha256:794fa942c2f593286370bbab2b6c6b75b9c4dcde84f62f522e59fb0f52ba05c1 size: 3033
[0] < Pushing echo-input done.
[0] worker done.

Finally, with your image pushed to Docker Hub, you can use it to deploy a function to your OpenFaaS server.

To deploy your function, run the deploy command, which takes the path to the manifest that describes your function, as well as the address of your OpenFaaS server:

faas-cli deploy -f echo-input.yml --gateway https://example.com

The output shows the status of the deployment, along with the name of the function you're deploying and the deployment status code:

Output
Deploying: echo-input.

Deployed. 200 OK.
URL: https://example.com/function/echo-input

If the deployment is successful, you will see a 200 status code. In the case of errors, follow the provided instructions to fix the problem before continuing.

At this point your function is deployed and ready to be used. You can test that it is working as expected by invoking it.

To invoke a function with the FaaS CLI, use the invoke command by passing the function name and OpenFaaS address to it. After executing the command, you'll be asked to enter the request you want to send to the function.

Execute the following command to invoke the echo-input function:

faas-cli invoke echo-input --gateway https://example.com

You'll be asked to enter the request you want to send to the function:

OutputReading from STDIN - hit (Control + D) to stop.

Enter the text you want to send to the function, such as:

Sammy The Shark!

Once you're done, press ENTER and then CTRL + D to finish the request. The CTRL + D shortcut in the terminal is used to register an End-of-File (EOF). The OpenFaaS CLI stops reading from the terminal once EOF is received.

After several seconds, the command will output the function's response:

OutputReading from STDIN - hit (Control + D) to stop.
Sammy The Shark!
Received message: Sammy The Shark!

If you don't see the output or you get an error, retrace the preceding steps to make sure you've deployed the function as explained and follow the on-screen instructions to resolve the problem.

At this point, you've interacted with your function using three methods: the Web UI, the API, and the CLI. Being able to execute your functions with any of these methods offers you the flexibility of deciding how you would like to integrate functions into your existing workflows.

Conclusion

In this tutorial, you've used serverless architecture and OpenFaaS to deploy and manage your applications using the OpenFaaS API, Web UI, and CLI. You also secured your infrastructure by leveraging Traefik to provide SSL using Let's Encrypt.

If you want to learn more about the OpenFaaS project, you can check out their website and the project's official documentation.

White Paper: Running Cloud Native Applications on DigitalOcean Kubernetes

2018-09-17T19:51:30Z

Download the Kubernetes White Paper

Running Cloud Native Applications on DigitalOcean Kubernetes (PDF)

Abstract

The Running Cloud Native Applications on DigitalOcean Kubernetes White Paper brings readers through a variety of cloud native topics, introducing them to how they may leverage Kubernetes in order to manage and scale their applications.

This white paper provides further insight into:

Trends in Modern Application Development
The Cloud Native Ecosystem
Microservices
Containers
Clusters
Kubernetes and DigitalOcean Kubernetes

Throughout the White Paper, a photo-sharing app called “Snappy” is used as a running example to demonstrate the value of implementing Cloud Native best practices.

Executive Summary: Scaling Cloud Native Apps

In today’s fast-moving software landscape, advances in operations technologies have fostered the dramatic reduction of application release cycles. Traditionally, software releases follow a time-based schedule, but it has become increasingly common to see applications and services continuously delivered and deployed to users throughout the day. This truncating of the traditional software release cycle has its roots both in technological developments — such as the explosive growth of cloud platforms, containers, and microservices-oriented architectures — as well as cultural developments — with tech-savvy and mobile-enabled users increasingly expecting new features, fast bug fixes, and a responsive and continuously developing product.

This symbiotic relationship between end users and developers has become increasingly linked. Shifting organizational structures and application architectures allow developers to quickly incorporate feedback and react to user demands. This accelerated development cadence often accompanies the packaging of applications into containers, and the use of systems that automate their deployment and orchestration, like Docker Swarm, Marathon, and Kubernetes. These open-source platforms, now stable enough for large-scale production deployments, allow service owners to launch and scale applications themselves, effortlessly managing hundreds of running containers.

Kubernetes and DigitalOcean Kubernetes

Kubernetes, initially open-sourced by Google in 2014, has today grown to become one of the highest velocity projects on GitHub, with over 11,300 contributing developers and 75,000 commits. The growth of its thriving open-source community mirrors its popularity in the private sector, with over 50% of Fortune 100 companies relying on Kubernetes every day to rapidly deploy new features and bug fixes to users.

DigitalOcean Kubernetes enables development teams both small and large to quickly take advantage of this market-leading container orchestration platform without the lead time required to provision, install, and operate a cluster. With its simplicity and developer-friendly interfaces, DigitalOcean Kubernetes empowers developers to launch their containerized applications into a managed, production-ready cluster without having to maintain and configure the underlying infrastructure. Seamlessly integrating with the rest of the DigitalOcean suite — including Load Balancers, Firewalls, Object Storage Spaces, and Block Storage Volumes — and with built-in support for public and private image registries like Docker Hub and Quay.io, developers can now run and scale container-based workloads with ease on the DigitalOcean platform.

With full programmatic control of their cluster using the exposed Kubernetes REST API, developers can benefit from the rich ecosystem of open-source tools while still reaping the convenience of managed infrastructure. Teams can flexibly deploy and scale their Cloud Native applications. A Certified Kubernetes conformant platform, DigitalOcean Kubernetes helps developers launch their application containers and bring their Kubernetes workloads into the DigitalOcean cloud with minimal configuration and operations overhead.

To learn more about scaling and managing Cloud Native applications, microservices, containers, and Kubernetes, download your free copy of Running Cloud Native Applications on DigitalOcean Kubernetes!

Download the Kubernetes White Paper

Running Cloud Native Applications on DigitalOcean Kubernetes (PDF)

Como Construir Imagens Docker e Hospedar um Repositório de Imagens Docker com o GitLab

2018-09-17T20:19:11Z

Como Construir Imagens Docker e Hospedar um Repositório de Imagens Docker com o GitLab

Introdução

A containerização está rapidamente se tornando o método de empacotamento e deploy de aplicações mais aceito nos ambientes de nuvem. A padronização que ele fornece, juntamente com sua eficiência de recursos (quando comparado a máquinas virtuais completas) e flexibilidade, o tornam um grande facilitador da moderna mentalidade DevOps. Muitas estratégias interessantes de deployment, orquestração e monitoramento nativas para nuvem tornam-se possíveis quando suas aplicações e microsserviços são totalmente containerizados.

Os containers Docker são de longe os tipos mais comuns de container atualmente. Embora os repositórios públicos de imagem do Docker como o Docker Hub estejam repletos de imagens de software opensource containerizado que você pode fazer um docker pull hoje, para código privado você precisará pagar um serviço para construir e armazenar suas imagens, ou executar seu próprio software para fazer isso.

O GitLab Community Edition é um pacote de software auto-hospedado que fornece hospedagem de repositório Git, acompanhamento de projetos, serviços de CI/CD, e um registro de imagem Docker, entre outros recursos. Neste tutorial vamos utilizar o serviço de integração contínua do GitLab para construir imagens Docker a partir de uma aplicação de exemplo em Node.js. Estas imagens serão então testadas e carregadas para o nosso próprio registro privado do Docker.

Pré-requisitos

Antes de começarmos, precisamos configurar um servidor GitLab seguro, e um GitLab CI runner para executar tarefas de integração contínua. As seções abaixo fornecerão links e maiores detalhes.

Um Servidor Gitlab Protegido com SSL

Para armazenar nosso código fonte, executar tarefas de CI/CD, e hospedar um registro Docker, precisamos de uma instância do GitLab instalada em um servidor Ubuntu 16.04. Atualmente, o GitLab recomenda um servidor com pelo menos 2 núcleos de CPU e 4GB de RAM. Adicionalmente, iremos proteger o servidor com certificados SSL do Let's Encrypt. Para fazer isto, precisaremos de um nome de domínio apontando para o servidor.

Você pode completar esses pré-requisitos com os seguintes tutoriais:

Como configurar um nome de host com a DigitalOcean mostrará como gerenciar um domínio com o painel de controle da DigitalOcean.
Configuração Inicial de servidor com Ubuntu 16.04 vai lhe fornecer um usuário não-root, habilitado para sudo, e habilitar o firewall ufw do Ubuntu.
Como instalar e configurar o GitLab no Ubuntu 16.04 irá lhe mostrar como instalar o GitLab e configurá-lo com um certificado TLS/SSL gratuito do Let's Encrypt

Um GitLab CI Runner

O tutorial Como configurar pipelines de integração contínua com o GitLab CI no Ubuntu 16.04 fornecerá uma visão geral do serviço de CI ou integração contínua do GitLab e mostrará como configurar um CI runner para processar jobs. Vamos construir isso em cima da aplicação de demonstração e da infraestrutura do runner criados neste tutorial.

Passo 1 — Configurando um GitLab CI Runner Privilegiado

No pré-requisito do tutorial de integração contínua com o GitLab, configuramos um GitLab runner utilizando sudo gitlab-runner register e seu processo de configuração interativo. Este runner é capaz de executar builds e testes de software dentro de containers Docker isolados.

Entretanto, para se construir imagens Docker, nosso runner precisa de acesso total ao próprio serviço do Docker. A maneira recomendada de se configurar isto é utilizar a imagem docker-in-docker oficial do Docker para executar os jobs. Isto requer conceder ao runner um modo de execução privileged ou privilegiado. Portanto, criaremos um segundo runner com este modo ativado.

Nota: Conceder ao runner o modo privileged basicamente desativa todas as vantagens de segurança da utilização de containers. Infelizmente, os outros métodos de ativar runners compatíveis com o Docker também carregam implicações de segurança semelhantes. Por favor, veja a documentação oficial do GitLab no Docker Build para aprender mais sobre as diferentes opções de runners e qual é a melhor para a sua situação.

Como existem implicações de segurança para a utilização de runner privilegiado, vamos criar um runner específico do projeto que aceitará somente jobs de Docker em nosso projeto hello_hapi (Os administradores de GitLab sempre podem adicionar manualmente esse runner a outros projetos posteriormente). A partir da página do nosso projeto hello_hapi, clique em Settings na parte inferior do menu à esquerda, em seguida clique em CI/CD no sub-menu:

Agora, clique no botão Expand ao lado da seção de configurações de Runners:

Haverá algumas informações sobre como configurar um Specific Runner, incluindo um token de registro. Tome nota desse token. Quando o utilizamos para registrar um novo runner, o runner será bloqueado apenas para este projeto.

Estando nesta página, clique no botão Disable shared Runners. Queremos ter certeza de que nossos jobs de Docker sempre executarão em nosso runner privilegiado. Se um runner compartilhado não privilegiado estivesse disponível, o GitLab pode optar por utilizá-lo, o que resultaria em erros de build.

Faça o login no servidor que possui o seu CI runner atual. Se você não tiver uma máquina já configurada com os runners, volte e complete a seção Instalando o Serviço CI Runner do GitLab do tutorial de pré-requisitos antes de continuar.

Agora, execute o seguinte comando para configurar o runner privilegiado específico do projeto:

sudo gitlab-runner register -n \
  --url https://gitlab.example.com/ \
  --registration-token seu-token \
  --executor docker \
  --description "docker-builder" \
  --docker-image "docker:latest" \
  --docker-privileged

OutputRegistering runner... succeeded                     runner=61SR6BwV
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Certifique-se de substituir suas próprias informações. Nós definimos todas as opções do nosso runner na linha de comando em vez de usar os prompts interativos, porque os prompts não nos permitem especificar o modo --docker-privileged.

Agora o seu runner está configurado, registrado e executando. Para verificar, volte ao seu navegador. Clique no ícone de chave inglesa na barra de menu principal do GitLab, em seguida clique em Runners no menu à esquerda. Seus runners serão listados:

Agora que temos um runner capaz de criar imagens do Docker, vamos configurar um registro privado do Docker para carregar imagens para ele.

Passo 2 — Configurando o Registro Docker do GitLab

Configurar seu próprio registro do Docker permite que você envie e extraia imagens de seu próprio servidor privado, aumentando a segurança e reduzindo as dependências do seu fluxo de trabalho em serviços externos.

O GitLab irá configurar um registro Docker privado com apenas algumas atualizações de configuração. Primeiro vamos configurar a URL onde o registro irá residir. Depois, iremos (opcionalmente) configurar o registro para usar um serviço de armazenamento de objetos compatível com S3 para armazenar seus dados.

Faça SSH em seu servidor GitLab, depois abra o arquivo de configuração do GitLab:

sudo nano /etc/gitlab/gitlab.rb

Role para baixo até a seção Container Registry settings. Vamos descomentar a linha registry_external_url e configurá-la para o nosso host GitLab com a porta número 5555:

/etc/gitlab/gitlab.rb


registry_external_url 'https://gitlab.example.com:5555'

A seguir, adicione as duas linhas seguintes para dizer ao registro onde encontrar nossos certificados Let's Encrypt:

/etc/gitlab/gitlab.rb


registry_nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab.example.com/fullchain.pem"
registry_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab.example.com/privkey.pem"

Salve e feche o arquivo, depois reconfigure o GitLab:

sudo gitlab-ctl reconfigure

Output. . .
gitlab Reconfigured!

Atualize o firewall para pemitir tráfego para a porta do registro:

sudo ufw allow 5555

Agora mude para outra máquina com o Docker instalado e efetue o login no registro Docker privado. Se você não tiver o Docker no seu computador de desenvolvimento local, você pode usar qualquer servidor configurado para executar seus jobs do GitLab CI, já que ele tem o Docker instalado:

docker login gitlab.example.com:5555

Você será solicitado para inserir o seu nome de usuário e senha. Use suas credenciais do GitLab para efetuar login.

OutputLogin Succeeded

Sucesso! O registro está configurado e funcionando. Atualmente, ele armazenará arquivos no sistema de arquivos local do servidor GitLab. Se você quiser usar um serviço de armazenamento de objetos, continue com esta seção. Se não, pule para o Passo 3.

Para configurar um backend de armazenamento de objetos para o registro, precisamos saber as seguintes informações sobre o nosso serviço de armazenamento de objetos:

Access Key
Secret Key
Region (us-east-1) por exemplo, se estiver usando Amazon S3, ou Region Endpoint se estiver usando um serviço compatível com S3 (https://nyc.digitaloceanspaces.com)
Nome do Bucket

Se você estiver usando o DigitalOcean Spaces, você pode descobrir como configurar um novo Space e obter as informações acima lendo Como Criar um Space e uma Chave de API na DigitalOcean.

Quando você tiver suas informações sobre o amazenamento de objetos, abra o arquivo de configuração do GitLab:

sudo nano /etc/gitlab/gitlab.rb

Novamente, role até a seção de registro do container. Procure pelo bloco registry['storage'], descomente o bloco e atualize-o para o seguinte, novamente certificando-se de substituir suas próprias informações, quando apropriado:

/etc/gitlab/gitlab.rb


registry['storage'] = {
  's3' => {
    'accesskey' => 'sua-key',
    'secretkey' => 'seu-secret',
    'bucket' => 'seu-bucket-name',
    'region' => 'nyc3',
    'regionendpoint' => 'https://nyc3.digitaloceanspaces.com'
  }
}

Se você estiver uando Amazon S3, você precisa apenas da region e não do regionendpoint. Se estiver usando um serviço S3 compatível, como o Spaces, você irá precisar do regionendpoint. Neste caso, region na verdade não configura nada e o valor que você digita não importa, mas ainda precisa estar presente e não em branco.

Salve e feche o arquivo.

Nota: Atualmente, há um bug em que o registro será encerrado após trinta segundos se seu bucket de armazenamento de objetos estiver vazio. Para evitar isso, coloque um arquivo no seu bucket antes de executar a próxima etapa. Você poderá removê-lo mais tarde, após o registro ter adicionado seus próprios objetos.

Se você estiver usando o Spaces da DigitalOcean, você pode arrastar e soltar um arquivo para carregá-lo usando a interface do Painel de Controle.

Reconfigure o GitLab mais uma vez:

sudo gitlab-ctl reconfigure

Em sua outra máquina Docker, efetue login no registro novamente para ter certeza de que tudo está bem:

docker login gitlab.example.com:5555

Você deve receber uma mensagem de Login Succeeded.

Agora que temos nosso registro do Docker configurado, vamos atualizar a configuração de CI da nossa aplicação para criar e testar nossa app, e enviar as imagens Docker para o nosso registro privado.

Passo 3 — Atualizando o `gitlab-ci.yaml` e Construindo uma Imagem Docker

Nota: Se você não concluiu o artigo de pré-requisito do GitLab CI você precisará copiar o repositório de exemplo para o seu servidor GitLab. Siga a seção Copiando o Repositório de Exemplo a partir do GitHub para fazer isto.

Para que possamos fazer o building de nossa app no Docker, precisamos atualizar o arquivo .gitlab-ci.yml. Você pode editar este arquivo diretamente no GitLab clicando na página principal do projeto, e depois no botão Edit. Alternativamente, você poderia clonar o repositório para a sua máquina local, editar o arquivo, e então fazer um git push nele de volta para o GitLab. Isso ficaria assim:

git clone git@gitlab.example.com:sammy/hello_hapi.git
cd hello_hapi
# edit the file w/ your favorite editor
git commit -am "updating ci configuration"
git push

Primeiro, exclua tudo no arquivo, depois cole nele a seguinte configuração:

.gitlab-ci.yml


image: docker:latest
services:
- docker:dind

stages:
- build
- test
- release

variables:
  TEST_IMAGE: gitlab.example.com:5555/sammy/hello_hapi:$CI_COMMIT_REF_NAME
  RELEASE_IMAGE: gitlab.example.com:5555/sammy/hello_hapi:latest

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.example.com:5555

build:
  stage: build
  script:
    - docker build --pull -t $TEST_IMAGE .
    - docker push $TEST_IMAGE

test:
  stage: test
  script:
    - docker pull $TEST_IMAGE
    - docker run $TEST_IMAGE npm test

release:
  stage: release
  script:
    - docker pull $TEST_IMAGE
    - docker tag $TEST_IMAGE $RELEASE_IMAGE
    - docker push $RELEASE_IMAGE
  only:
    - master

Certifique-se de atualizar os URLs e nomes de usuários realçados com suas próprias informações e, em seguida, salve com o botão Commit changes no GitLab. Se você está atualizando o arquivo fora do GitLab, confirme as mudanças e faça git push de volta no GitLab.

Este novo arquivo de configuração diz ao GitLab para usar a imagem mais recente do docker (image: docker:latest) e vinculá-la ao serviço docker-in-docker (docker:dind). Então, ele define os estágios de build, test, e release. O estágio de build cria a imagem do Docker usando o Dockerfile fornecido pelo repositório, em seguida o carrega para o nosso registro de imagens Docker. Se isso for bem sucedido, o estágio test vai baixar a imagem que acabamos de construir e executar o comando npm test dentro dele. Se o estágio test for bem sucedido, o estágio release irá lançar a imagem, irá colocar uma tag como hello_hapi:latest e irá retorná-la ao registro.

Dependendo do seu fluxo de trabalho, você também pode adicionar mais estágios test, ou mesmo estágios deploy que levam o aplicativo para um ambiente de preparação ou produção.

A atualização do arquivo de configuração deve ter acionado um novo build. Volte ao projeto hello_hapi no GitLab e clique no indicador de status do CI para o commit:

Na página resultante, você pode clicar em qualquer um dos estágios para ver seu progresso:

Eventualmente, todas as etapas devem indicar que eles foram bem sucedidos, mostrando ícones com a marca de verificação em verde. Podemos encontrar as imagens Docker que acabaram de ser construídas clicando no item Registry no menu à esquerda:

Se você clicar no pequeno ícone "document" ao lado do nome da imagem, ele copiará o comando apropriado docker pull ... para a sua área de transferência. Você pode então baixar e executar sua imagem:

docker pull gitlab.example.com:5555/sammy/hello_hapi:latest
docker run -it --rm -p 3000:3000 gitlab.example.com:5555/sammy/hello_hapi:latest

Output> hello@1.0.0 start /usr/src/app
> node app.js

Server running at: http://56fd5df5ddd3:3000

A imagem foi baixada do registro e iniciada em um container. Mude para o seu navegador e conecte-se ao aplicativo na porta 3000 para testar. Neste caso, estamos executando o container em nossa máquina local, assim podemos acessá-la via localhost na seguinte URL:

http://localhost:3000/hello/test

OutputHello, test!

Sucesso! Você pode parar o container com CTRL-C. A partir de agora, toda vez que enviarmos um novo código para a ramificação master do nosso repositório, vamos construir e testar automaticamente uma nova imagem hello_hapi: latest.

Conclusão

Neste tutorial, configuramos um novo GitLab runner para criar imagens do Docker, criamos um regisro privado do Docker para armazená-las, e atualizamos um app Node.js para ser construído e testado dentro de containers Docker.

Para aprender mais sobre os vários componentes utilizados nesta configuração, você pode ler a documentação oficial do GitLab CE, GitLab Container Registry, e do Docker.

Por Brian Boucheron

How To Install WordPress with LEMP on Debian 9

2018-09-14T16:28:22Z

Introduction

WordPress is the most popular CMS (content management system) on the internet. It allows you to easily set up flexible blogs and websites on top of a MySQL backend with PHP processing. WordPress has seen incredible adoption and is a great choice for getting a website up and running quickly. After setup, almost all administration can be done through the web frontend.

In this guide, we'll focus on getting a WordPress instance set up on a LEMP stack (Linux, Nginx, MySQL, and PHP) on a Debian 9 server.

Prerequisites

In order to complete this tutorial, you will need access to a Debian 9 server.

You will need to perform the following tasks before you can start this guide:

Create a sudo user on your server: We will be completing the steps in this guide using a non-root user with sudo privileges. You can create a user with sudo privileges by following our Debian 9 initial server setup guide.
Install a LEMP stack: WordPress will need a web server, a database, and PHP in order to correctly function. Setting up a LEMP stack (Linux, Nginx, MySQL, and PHP) fulfills all of these requirements. Follow this guide to install and configure this software.
Secure your site with SSL: WordPress serves dynamic content and handles user authentication and authorization. TLS/SSL is the technology that allows you to encrypt the traffic from your site so that your connection is secure. This tutorial will assume that you have a domain name for your blog. You can use Let's Encrypt to get a free SSL certificate for your domain. Follow our Let's Encrypt guide for Nginx to set this up.

When you are finished the setup steps, log into your server as your sudo user and continue below.

Step 1 — Creating a MySQL Database and User for WordPress

The first step that we will take is a preparatory one. WordPress uses MySQL to manage and store site and user information. We have MySQL installed already, but we need to make a database and a user for WordPress to use.

To get started, log into the MySQL root (administrative) account. If MySQL is configured to use the auth_socket authentication plugin (the default), you can log into the MySQL administrative account using sudo:

sudo mysql

If you changed the authentication method to use a password for the MySQL root account, use the following format instead:

mysql -u root -p

You will be prompted for the password you set for the MySQL root account.

First, we can create a separate database that WordPress can control. You can call this whatever you would like, but we will be using wordpress in this guide to keep it simple. You can create the database for WordPress by typing:

CREATE DATABASE your_domain DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Note: Every MySQL statement must end in a semi-colon (;). Check to make sure this is present if you are running into any issues.

Next, we are going to create a separate MySQL user account that we will use exclusively to operate on our new database. Creating one-function databases and accounts is a good idea from a management and security standpoint. We will use the name wordpressuser in this guide. Feel free to change this if you'd like.

We are going to create this account, set a password, and grant access to the database we created. We can do this by typing the following command. Remember to choose a strong password here for your database user:

GRANT ALL ON your_domain.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password';

You now have a database and user account, each made specifically for WordPress. We need to flush the privileges so that the current instance of MySQL knows about the recent changes we've made:

FLUSH PRIVILEGES;

Exit out of MySQL by typing:

EXIT;

The MySQL session will exit, returning you to the regular Linux shell.

Step 2 — Installing Additional PHP Extensions

When setting up our LEMP stack, we only required a very minimal set of extensions in order to get PHP to communicate with MySQL. WordPress and many of its plugins leverage additional PHP extensions.

We can download and install some of the most popular PHP extensions for use with WordPress by typing:

sudo apt update
sudo apt install php-curl php-gd php-intl php-mbstring php-soap php-xml php-xmlrpc php-zip

Note: Each WordPress plugin has its own set of requirements. Some may require additional PHP packages to be installed. Check your plugin documentation to discover its PHP requirements. If they are available, they can be installed with apt as demonstrated above.

When you are finished installing the extensions, restart the PHP-FPM process so that the running PHP processor can leverage the newly installed features:

sudo systemctl restart php7.0-fpm

We now have all of the necessary PHP extensions installed on the server.

Step 3 — Configuring Nginx

Next, we will be making a few minor adjustments to our Nginx server block files. Based on the prerequisite tutorials, you should have a configuration file for your site in the /etc/nginx/sites-available/ directory configured to respond to your server's domain name and protected by a TLS/SSL certificate. We'll use /etc/nginx/sites-available/your_domain as an example here, but you should substitute the path to your configuration file where appropriate.

Additionally, we will use /var/www/your_domain as the root directory of our WordPress install. You should use the web root specified in your own configuration.

Note: It's possible you are using the /etc/nginx/sites-available/default default configuration (with /var/www/html as your web root). This is fine to use if you're only going to host one website on this server. If not, it's best to split the necessary configuration into logical chunks, one file per site.

Open your site's Nginx configuration file with sudo privileges to begin:

sudo nano /etc/nginx/sites-available/your_domain

We need to add a few location directives within our main server block. After adding SSL certificates your config may have two server blocks. If so, find the one that contains root /var/www/your_domain and your other location directives and implement your changes there.

Start by creating exact-matching location blocks for requests to /favicon.ico and /robots.txt, both of which we do not want to log requests for.

We will use a regular expression location to match any requests for static files. We will again turn off the logging for these requests and will mark them as highly cacheable since these are typically expensive resources to serve. You can adjust this static files list to contain any other file extensions your site may use:

/etc/nginx/sites-available/your_domain

server {
    . . .

    location = /favicon.ico { log_not_found off; access_log off; }
    location = /robots.txt { log_not_found off; access_log off; allow all; }
    location ~* \.(css|gif|ico|jpeg|jpg|js|png)$ {
        expires max;
        log_not_found off;
    }
    . . .
}

Inside of the existing location / block, we need to adjust the try_files list so that instead of returning a 404 error as the default option, control is passed to the index.php file with the request arguments.

This should look something like this:

/etc/nginx/sites-available/wordpress

server {
    . . .
    location / {
        #try_files $uri $uri/ =404;
        try_files $uri $uri/ /index.php$is_args$args;
    }
    . . .
}

When you are finished, save and close the file.

Now, we can check our configuration for syntax errors by typing:

sudo nginx -t

If no errors were reported, reload Nginx by typing:

sudo systemctl reload nginx

Next, we will download and set up WordPress itself.

Step 4 — Downloading WordPress

Now that our server software is configured, we can download and set up WordPress. For security reasons in particular, it is always recommended to get the latest version of WordPress from their site.

Change into a writable directory and then download the compressed release by typing:

cd /tmp
curl -LO https://wordpress.org/latest.tar.gz

Extract the compressed file to create the WordPress directory structure:

tar xzvf latest.tar.gz

We will be moving these files into our document root momentarily. Before we do that, we can copy over the sample configuration file to the filename that WordPress actually reads:

cp /tmp/wordpress/wp-config-sample.php /tmp/wordpress/wp-config.php

Now, we can copy the entire contents of the directory into our document root. We are using the -a flag to make sure our permissions are maintained. We are using a dot at the end of our source directory to indicate that everything within the directory should be copied, including any hidden files:

sudo cp -a /tmp/wordpress/. /var/www/your_domain

Now that our files are in place, we'll assign ownership them to the www-data user and group. This is the user and group that Nginx runs as, and Nginx will need to be able to read and write WordPress files in order to serve the website and perform automatic updates.

sudo chown -R www-data:www-data /var/www/your_domain

Our files are now in our server's document root and have the correct ownership, but we still need to complete some more configuration.

Step 5 — Setting up the WordPress Configuration File

Next, we need to make some changes to the main WordPress configuration file.

When we open the file, our first order of business will be to adjust some secret keys to provide some security for our installation. WordPress provides a secure generator for these values so that you do not have to try to come up with good values on your own. These are only used internally, so it won't hurt usability to have complex, secure values here.

To grab secure values from the WordPress secret key generator, type:

curl -s https://api.wordpress.org/secret-key/1.1/salt/

You will get back unique values that look something like this:

Warning: It is important that you request unique values each time. Do NOT copy the values shown below!

Output
define('AUTH_KEY',         '1jl/vqfs<XhdXoAPz9 DO NOT COPY THESE VALUES c_j{iwqD^<+c9.k<J@4H');
define('SECURE_AUTH_KEY',  'E2N-h2]Dcvp+aS/p7X DO NOT COPY THESE VALUES {Ka(f;rv?Pxf})CgLi-3');
define('LOGGED_IN_KEY',    'W(50,{W^,OPB%PB<JF DO NOT COPY THESE VALUES 2;y&,2m%3]R6DUth[;88');
define('NONCE_KEY',        'll,4UC)7ua+8<!4VM+ DO NOT COPY THESE VALUES #`DXF+[$atzM7 o^-C7g');
define('AUTH_SALT',        'koMrurzOA+|L_lG}kf DO NOT COPY THESE VALUES  07VC*Lj*lD&?3w!BT#-');
define('SECURE_AUTH_SALT', 'p32*p,]z%LZ+pAu:VY DO NOT COPY THESE VALUES C-?y+K0DK_+F|0h{!_xY');
define('LOGGED_IN_SALT',   'i^/G2W7!-1H2OQ+t$3 DO NOT COPY THESE VALUES t6**bRVFSD[Hi])-qS`|');
define('NONCE_SALT',       'Q6]U:K?j4L%Z]}h^q7 DO NOT COPY THESE VALUES 1% ^qUswWgn+6&xqHN&%');

These are configuration lines that we can paste directly in our configuration file to set secure keys. Copy the output you received now.

Now, open the WordPress configuration file:

sudo nano /var/www/your_domain/wp-config.php

Find the section that contains the dummy values for those settings. It will look something like this:

/var/www/wordpress/wp-config.php

. . .

define('AUTH_KEY',         'put your unique phrase here');
define('SECURE_AUTH_KEY',  'put your unique phrase here');
define('LOGGED_IN_KEY',    'put your unique phrase here');
define('NONCE_KEY',        'put your unique phrase here');
define('AUTH_SALT',        'put your unique phrase here');
define('SECURE_AUTH_SALT', 'put your unique phrase here');
define('LOGGED_IN_SALT',   'put your unique phrase here');
define('NONCE_SALT',       'put your unique phrase here');

. . .

Delete those lines and paste in the values you copied from the command line:

/var/www/wordpress/wp-config.php

. . .

define('AUTH_KEY',         'VALUES COPIED FROM THE COMMAND LINE');
define('SECURE_AUTH_KEY',  'VALUES COPIED FROM THE COMMAND LINE');
define('LOGGED_IN_KEY',    'VALUES COPIED FROM THE COMMAND LINE');
define('NONCE_KEY',        'VALUES COPIED FROM THE COMMAND LINE');
define('AUTH_SALT',        'VALUES COPIED FROM THE COMMAND LINE');
define('SECURE_AUTH_SALT', 'VALUES COPIED FROM THE COMMAND LINE');
define('LOGGED_IN_SALT',   'VALUES COPIED FROM THE COMMAND LINE');
define('NONCE_SALT',       'VALUES COPIED FROM THE COMMAND LINE');

. . .

Next, we need to modify some of the database connection settings at the beginning of the file. You need to adjust the database name, the database user, and the associated password that we configured within MySQL.

The other change we need to make is to set the method that WordPress should use to write to the filesystem. Since we've given the web server permission to write where it needs to, we can explicitly set the filesystem method to "direct". Failure to set this with our current settings would result in WordPress prompting for FTP credentials when we perform some actions. This setting can be added below the database connection settings, or anywhere else in the file:

/var/www/wordpress/wp-config.php

. . .

define('DB_NAME', 'your_domain');

/** MySQL database username */
define('DB_USER', 'wordpressuser');

/** MySQL database password */
define('DB_PASSWORD', 'password');

. . .

define('FS_METHOD', 'direct');

Save and close the file when you are finished.

Step 6 — Completing the Installation Through the Web Interface

Now that the server configuration is complete, we can finish up the installation through the web interface.

In your web browser, navigate to your server's domain name or public IP address:

http://server_domain_or_IP

Select the language you would like to use:

Next, you will come to the main setup page.

Select a name for your WordPress site and choose a username (it is recommended not to choose something like "admin" for security purposes). A strong password is generated automatically. Save this password or select an alternative strong password.

Enter your email address and select whether you want to discourage search engines from indexing your site:

When you click ahead, you will be taken to a page that prompts you to log in:

Once you log in, you will be taken to the WordPress administration dashboard:

Conclusion

WordPress should be installed and ready to use! Some common next steps are to choose the permalinks setting for your posts (can be found in Settings > Permalinks) or to select a new theme (in Appearance > Themes). If this is your first time using WordPress, explore the interface a bit to get acquainted with your new CMS.

How To Install Linux, Nginx, MySQL, PHP (LEMP stack) on Debian 9

2018-09-13T19:52:25Z

Introduction

The LEMP software stack is a group of software that can be used to serve dynamic web pages and web applications. This is an acronym that describes a Linux operating system, with an Nginx web server. The backend data is stored in the MySQL database and the dynamic processing is handled by PHP.

In this guide, you'll install a LEMP stack on a Debian server using the packages provided by the operating system.

Prerequisites

To complete this guide, you will need a Debian 9 server with a non-root user with sudo privileges. You can set up a user with these privileges in our Initial Server Setup with Debian 9 guide.

Step 1 — Installing the Nginx Web Server

In order to display web pages to our site visitors, we are going to employ Nginx, a modern, efficient web server.

All of the software we will be using for this procedure will come directly from Debian's default package repositories. This means we can use the apt package management suite to complete the installation.

Since this is our first time using apt for this session, we should start off by updating our local package index. We can then install the server:

sudo apt update
sudo apt install nginx

On Debian 9, Nginx is configured to start running upon installation.

If you have the ufw firewall running, you will need to allow connections to Nginx. You should enable the most restrictive profile that will still allow the traffic you want. Since we haven't configured SSL for our server yet, in this guide, we will only need to allow traffic on port 80.

You can enable this by typing:

sudo ufw allow 'Nginx HTTP'

You can verify the change by typing:

sudo ufw status

You should see HTTP traffic allowed in the displayed output:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx HTTP                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

Now, test if the server is up and running by accessing your server's domain name or public IP address in your web browser. If you do not have a domain name pointed at your server and you do not know your server's public IP address, you can find it by typing one of the following into your terminal:

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'

This will print out a few IP addresses. You can try each of them in turn in your web browser.

Type one of the addresses that you receive in your web browser. It should take you to Nginx's default landing page:

http://your_domain_or_IP

If you see the above page, you have successfully installed Nginx.

Step 2 — Installing MySQL to Manage Site Data

Now that we have a web server, we need to install MySQL, a database management system, to store and manage the data for our site.

You can install this easily by typing:

sudo apt install mysql-server

Note: In Debian 9 a community fork of the MySQL project – MariaDB – is packaged as the default MySQL variant. While, MariaDB works well in most cases, if you need features found only in Oracle's MySQL, you can install and use packages from a repository maintained by the MySQL developers. To install the official MySQL server, use our tutorial How To Install the Latest MySQL on Debian 9.

The MySQL database software is now installed, but its configuration is not complete.

To secure the installation, we can run a security script that will ask whether we want to modify some insecure defaults. Begin the script by typing:

sudo mysql_secure_installation

You will be asked to enter the password for the MySQL root account. We haven't set this yet, so just hit ENTER. Then you'll be asked you if you want to set that password. You should type y then set a root password.

For the rest of the questions the script asks, you should press y, followed by the ENTER key at each prompt. This will remove some anonymous users and the test database, disable remote root logins, and load these new rules so that MySQL immediately respects the changes you have made.

At this point, your database system is now set up and secured. Let's set up PHP.

Step 3 — Installing PHP for Processing

We now have Nginx installed to serve our pages and MySQL installed to store and manage our data. However, we still don't have anything that can generate dynamic content. That's where PHP comes in.

Since Nginx does not contain native PHP processing like some other web servers, we will need to install fpm, which stands for "fastCGI process manager". We will tell Nginx to pass PHP requests to this software for processing. We'll also install an additional helper package that will allow PHP to communicate with our MySQL database backend. The installation will pull in the necessary PHP core files to make that work.

Then install the php-fpm and php-mysql packages:

sudo apt install php-fpm php-mysql

We now have our PHP components installed. Next we'll configure Nginx to use them.

Step 4 — Configuring Nginx to Use the PHP Processor

Now we have all of the required components installed. The only configuration change we still need is to tell Nginx to use our PHP processor for dynamic content.

We do this on the server block level (server blocks are similar to Apache's virtual hosts). We're going to leave the default Nginx configuration alone and instead create a new configuration file and new web root directory to hold our PHP files. We'll name the configuration file and the directory after the domain name or hostname that the server should respond to.

First, create a new directory in /var/www to hold the PHP site:

sudo mkdir /var/www/your_domain

Then, open a new configuration file in Nginx's sites-available directory:

sudo nano /etc/nginx/sites-available/your_domain

This will create a new blank file. Paste in the following bare-bones configuration:

/etc/nginx/sites-available/your_domain

server {
    listen 80;
    listen [::]:80;

    root /var/www/your_domain;
    index index.php index.html index.htm;

    server_name your_domain;

    location / {
        try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
    }
}

This is a very basic configuration that listens on port 80 and serves files from the web root we just created. It will only respond to requests to the name provided after server_name, and any files ending in .php will be processed by the php-fpm process before Nginx sends the results to the user.

Save and close the file when you're done customizing it.

Activate your configuration by linking to the config file from Nginx's sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/your_domain.conf /etc/nginx/sites-enabled/

This will tell Nginx to use the configuration next time it is reloaded. First, test your configuration for syntax errors by typing:

sudo nginx -t

If any errors are reported, go back and recheck your file before continuing.

When you are ready, reload Nginx to make the changes:

sudo systemctl reload nginx

Next we'll create a file in our new web root directory to test out PHP processing.

Step 5 — Create a PHP File to Test Configuration

Your LEMP stack should now be completely set up. We can test it to validate that Nginx can correctly hand .php files off to our PHP processor.

We can do this by creating a test PHP file in our document root. Open a new file called info.php within your document root in your text editor:

sudo nano /var/www/your_domain/info.php

Type or paste the following lines into the new file. This is valid PHP code that will return information about our server:

/var/www/your_domain/info.php

<?php
  phpinfo();
?>

When you are finished, save and close the file.

Now, you can visit this page in your web browser by visiting your server's domain name or public IP address followed by /info.php:

http://your_domain/info.php

You should see a web page that has been generated by PHP with information about your server:

If you see a page that looks like this, you've set up PHP processing with Nginx successfully.

After verifying that Nginx renders the page correctly, it's best to remove the file you created as it can actually give unauthorized users some hints about your configuration that may help them try to break in.

For now, remove the file by typing:

sudo rm /var/www/html/info.php

You can always regenerate this file if you need it later.

Conclusion

You should now have a LEMP stack configured on your Debian server. This gives you a very flexible foundation for serving web content to your visitors.

Modernizing Applications for Kubernetes

2018-09-11T22:07:45Z

Introduction

Modern stateless applications are built and designed to run in software containers like Docker, and be managed by container clusters like Kubernetes. They are developed using Cloud Native and Twelve Factor principles and patterns, to minimize manual intervention and maximize portability and redundancy. Migrating virtual-machine or bare metal-based applications into containers (known as "containerizing") and deploying them inside of clusters often involves significant shifts in how these apps are built, packaged, and delivered.

Building on Architecting Applications for Kubernetes, in this conceptual guide, we'll discuss high-level steps for modernizing your applications, with the end goal of running and managing them in a Kubernetes cluster. Although you can run stateful applications like databases on Kubernetes, this guide focuses on migrating and modernizing stateless applications, with persistent data offloaded to an external data store. Kubernetes provides advanced functionality for efficiently managing and scaling stateless applications, and we'll explore the application and infrastructure changes necessary for running scalable, observable, and portable apps on Kubernetes.

Preparing the Application for Migration

Before containerizing your application or writing Kubernetes Pod and Deployment configuration files, you should implement application-level changes to maximize your app's portability and observability in Kubernetes. Kubernetes is a highly automated environment that can automatically deploy and restart failing application containers, so it's important to build in the appropriate application logic to communicate with the container orchestrator and allow it to automatically scale your app as necessary.

Extract Configuration Data

One of the first application-level changes to implement is extracting application configuration from application code. Configuration consists of any information that varies across deployments and environments, like service endpoints, database addresses, credentials, and various parameters and options. For example, if you have two environments, say staging and production, and each contains a separate database, your application should not have the database endpoint and credentials explicitly declared in the code, but stored in a separate location, either as variables in the running environment, a local file, or external key-value store, from which the values are read into the app.

Hardcoding these parameters into your code poses a security risk as this config data often consists of sensitive information, which you then check in to your version control system. It also increases complexity as you now have to maintain multiple versions of your application, each consisting of the same core application logic, but varying slightly in configuration. As applications and their configuration data grow, hardcoding config into app code quickly becomes unwieldy.

By extracting configuration values from your application code, and instead ingesting them from the running environment or local files, your app becomes a generic, portable package that can be deployed into any environment, provided you supply it with accompanying configuration data. Container software like Docker and cluster software like Kubernetes have been designed around this paradigm, building in features for managing configuration data and injecting it into application containers. These features will be covered in more detail in the Containerizing and Kubernetes sections.

Here’s a quick example demonstrating how to externalize two config values DB_HOST and DB_USER from a simple Python Flask app’s code. We'll make them available in the app’s running environment as env vars, from which the app will read them:

hardcoded_config.py

from flask import Flask

DB_HOST = 'mydb.mycloud.com'
DB_USER = 'sammy'

app = Flask(__name__)

@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

Running this simple app (consult the Flask Quickstart to learn how) and visiting its web endpoint will display a page containing these two config values.

Now, here’s the same example with the config values externalized to the app’s running environment:

env_config.py

import os

from flask import Flask

DB_HOST = os.environ.get('APP_DB_HOST')
DB_USER = os.environ.get('APP_DB_USER')

app = Flask(__name__)

@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

Before running the app, we set the necessary config variables in the local environment:

export APP_DB_HOST=mydb.mycloud.com
export APP_DB_USER=sammy
flask run

The displayed web page should contain the same text as in the first example, but the app’s config can now be modified independently of the application code. You can use a similar approach to read in config parameters from a local file.

In the next section we’ll discuss moving application state outside of containers.

Offload Application State

Cloud Native applications run in containers, and are dynamically orchestrated by cluster software like Kubernetes or Docker Swarm. A given app or service can be load balanced across multiple replicas, and any individual app container should be able to fail, with minimal or no disruption of service for clients. To enable this horizontal, redundant scaling, applications must be designed in a stateless fashion. This means that they respond to client requests without storing persistent client and application data locally, and at any point in time if the running app container is destroyed or restarted, critical data is not lost.

For example, if you are running an address book application and your app adds, removes and modifies contacts from an address book, the address book data store should be an external database or other data store, and the only data kept in container memory should be short-term in nature, and disposable without critical loss of information. Data that persists across user visits like sessions should also be moved to external data stores like Redis. Wherever possible, you should offload any state from your app to services like managed databases or caches.

For stateful applications that require a persistent data store (like a replicated MySQL database), Kubernetes builds in features for attaching persistent block storage volumes to containers and Pods. To ensure that a Pod can maintain state and access the same persistent volume after a restart, the StatefulSet workload must be used. StatefulSets are ideal for deploying databases and other long-running data stores to Kubernetes.

Stateless containers enable maximum portability and full use of available cloud resources, allowing the Kubernetes scheduler to quickly scale your app up and down and launch Pods wherever resources are available. If you don’t require the stability and ordering guarantees provided by the StatefulSet workload, you should use the Deployment workload to manage and scale and your applications.

To learn more about the design and architecture of stateless, Cloud Native microservices, consult our Kubernetes White Paper.

Implement Health Checks

In the Kubernetes model, the cluster control plane can be relied on to repair a broken application or service. It does this by checking the health of application Pods, and restarting or rescheduling unhealthy or unresponsive containers. By default, if your application container is running, Kubernetes sees your Pod as "healthy." In many cases this is a reliable indicator for the health of a running application. However, if your application is deadlocked and not performing any meaningful work, the app process and container will continue to run indefinitely, and by default Kubernetes will keep the stalled container alive.

To properly communicate application health to the Kubernetes control plane, you should implement custom application health checks that indicate when an application is both running and ready to receive traffic. The first type of health check is called a readiness probe, and lets Kubernetes know when your application is ready to receive traffic. The second type of check is called a liveness probe, and lets Kubernetes know when your application is healthy and running. The Kubelet Node agent can perform these probes on running Pods using 3 different methods:

HTTP: The Kubelet probe performs an HTTP GET request against an endpoint (like /health), and succeeds if the response status is between 200 and 399
Container Command: The Kubelet probe executes a command inside of the running container. If the exit code is 0, then the probe succeeds.
TCP: The Kubelet probe attempts to connect to your container on a specified port. If it can establish a TCP connection, then the probe succeeds.

You should choose the appropriate method depending on the running application(s), programming language, and framework. The readiness and liveness probes can both use the same probe method and perform the same check, but the inclusion of a readiness probe will ensure that the Pod doesn't receive traffic until the probe begins succeeding.

When planning and thinking about containerizing your application and running it on Kubernetes, you should allocate planning time for defining what "healthy" and "ready" mean for your particular application, and development time for implementing and testing the endpoints and/or check commands.

Here’s a minimal health endpoint for the Flask example referenced above:

env_config.py

. . .  
@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

@app.route('/health')
def return_ok():
    return 'Ok!', 200

A Kubernetes liveness probe that checks this path would then look something like this:

pod_spec.yaml

. . .
  livenessProbe:
      httpGet:
        path: /health
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 2

The initialDelaySeconds field specifies that Kubernetes (specifically the Node Kubelet) should probe the /health endpoint after waiting 5 seconds, and periodSeconds tells the Kubelet to probe /health every 2 seconds.

To learn more about liveness and readiness probes, consult the Kubernetes documentation.

Instrument Code for Logging and Monitoring

When running your containerized application in an environment like Kubernetes, it's important to publish telemetry and logging data to monitor and debug your application's performance. Building in features to publish performance metrics like response duration and error rates will help you monitor your application and alert you when your application is unhealthy.

One tool you can use to monitor your services is Prometheus, an open-source systems monitoring and alerting toolkit, hosted by the Cloud Native Computing Foundation (CNCF). Prometheus provides several client libraries for instrumenting your code with various metric types to count events and their durations. For example, if you're using the Flask Python framework, you can use the Prometheus Python client to add decorators to your request processing functions to track the time spent processing requests. These metrics can then be scraped by Prometheus at an HTTP endpoint like /metrics.

A helpful method to use when designing your app's instrumentation is the RED method. It consists of the following three key request metrics:

Rate: The number of requests received by your application
Errors: The number of errors emitted by your application
Duration: The amount of time it takes your application to serve a response

This minimal set of metrics should give you enough data to alert on when your application's performance degrades. Implementing this instrumentation along with the health checks discussed above will allow you to quickly detect and recover from a failing application.

To learn more about signals to measure when monitoring your applications, consult Monitoring Distributed Systems from the Google Site Reliability Engineering book.

In addition to thinking about and designing features for publishing telemetry data, you should also plan how your application will log in a distributed cluster-based environment. You should ideally remove hardcoded configuration references to local log files and log directories, and instead log directly to stdout and stderr. You should treat logs as a continuous event stream, or sequence of time-ordered events. This output stream will then get captured by the container enveloping your application, from which it can be forwarded to a logging layer like the EFK (Elasticsearch, Fluentd, and Kibana) stack. Kubernetes provides a lot of flexibility in designing your logging architecture, which we'll explore in more detail below.

Build Administration Logic into API

Once your application is containerized and up and running in a cluster environment like Kubernetes, you may no longer have shell access to the container running your app. If you've implemented adequate health checking, logging, and monitoring, you can quickly be alerted on, and debug production issues, but taking action beyond restarting and redeploying containers may be difficult. For quick operational and maintenance fixes like flushing queues or clearing a cache, you should implement the appropriate API endpoints so that you can perform these operations without having to restart containers or exec into running containers and execute series of commands. Containers should be treated as immutable objects, and manual administration should be avoided in a production environment. If you must perform one-off administrative tasks, like clearing caches, you should expose this functionality via the API.

Summary

In these sections we’ve discussed application-level changes you may wish to implement before containerizing your application and moving it to Kubernetes. For a more in-depth walkthrough on building Cloud Native apps, consult Architecting Applications for Kubernetes.

We’ll now discuss some considerations to keep in mind when building containers for your apps.

Containerizing Your Application

Now that you've implemented app logic to maximize its portability and observability in a cloud-based environment, it's time to package your app inside of a container. For the purposes of this guide, we'll use Docker containers, but you should use whichever container implementation best suits your production needs.

Explicitly Declare Dependencies

Before creating a Dockerfile for your application, one of the first steps is taking stock of the software and operating system dependencies your application needs to run correctly. Dockerfiles allow you to explicitly version every piece of software installed into the image, and you should take advantage of this feature by explicitly declaring the parent image, software library, and programming language versions.

Avoid latest tags and unversioned packages as much as possible, as these can shift, potentially breaking your application. You may wish to create a private registry or private mirror of a public registry to exert more control over image versioning and to prevent upstream changes from unintentionally breaking your image builds.

To learn more about setting up a private image registry, consult Deploy a Registry Server from the Docker official documentation and the Registries section below.

Keep Image Sizes Small

When deploying and pulling container images, large images can significantly slow things down and add to your bandwidth costs. Packaging a minimal set of tools and application files into an image provides several benefits:

Reduce image sizes
Speed up image builds
Reduce container start lag
Speed up image transfer times
Improve security by reducing attack surface

Some steps you can consider when building your images:

Use a minimal base OS image like alpine or build from scratch instead of a fully featured OS like ubuntu
Clean up unnecessary files and artifacts after installing software
Use separate "build" and "runtime" containers to keep production application containers small
Ignore unnecessary build artifacts and files when copying in large directories

For a full guide on optimizing Docker containers, including many illustrative examples, consult Building Optimized Containers for Kubernetes.

Inject Configuration

Docker provides several helpful features for injecting configuration data into your app's running environment.

One option for doing this is specifying environment variables and their values in the Dockerfile using the ENV statement, so that configuration data is built-in to images:

Dockerfile

...
ENV MYSQL_USER=my_db_user
...

Your app can then parse these values from its running environment and configure its settings appropriately.

You can also pass in environment variables as parameters when starting a container using docker run and the -e flag:

docker run -e MYSQL_USER='my_db_user' IMAGE[:TAG]

Finally, you can use an env file, containing a list of environment variables and their values. To do this, create the file and use the --env-file parameter to pass it in to the command:

docker run --env-file var_list IMAGE[:TAG]

If you're modernizing your application to run it using a cluster manager like Kubernetes, you should further externalize your config from the image, and manage configuration using Kubernetes' built-in ConfigMap and Secrets objects. This allows you to separate configuration from image manifests, so that you can manage and version it separately from your application. To learn how to externalize configuration using ConfigMaps and Secrets, consult the ConfigMaps and Secrets section below.

Publish Image to a Registry

Once you've built your application images, to make them available to Kubernetes, you should upload them to a container image registry. Public registries like Docker Hub host the latest Docker images for popular open source projects like Node.js and nginx. Private registries allow you publish your internal application images, making them available to developers and infrastructure, but not the wider world.

You can deploy a private registry using your existing infrastructure (e.g. on top of cloud object storage), or optionally use one of several Docker registry products like Quay.io or paid Docker Hub plans. These registries can integrate with hosted version control services like GitHub so that when a Dockerfile is updated and pushed, the registry service will automatically pull the new Dockerfile, build the container image, and make the updated image available to your services.

To exert more control over the building and testing of your container images and their tagging and publishing, you can implement a continuous integration (CI) pipeline.

Implement a Build Pipeline

Building, testing, publishing and deploying your images into production manually can be error-prone and does not scale well. To manage builds and continuously publish containers containing your latest code changes to your image registry, you should use a build pipeline.

Most build pipelines perform the following core functions:

Watch source code repositories for changes
Run smoke and unit tests on modified code
Build container images containing modified code
Run further integration tests using built container images
If tests pass, tag and publish images to registry
(Optional, in continuous deployment setups) Update Kubernetes Deployments and roll out images to staging/production clusters

There are many paid continuous integration products that have built-in integrations with popular version control services like GitHub and image registries like Docker Hub. An alternative to these products is Jenkins, a free and open-source build automation server that can be configured to perform all of the functions described above. To learn how to set up a Jenkins continuous integration pipeline, consult How To Set Up Continuous Integration Pipelines in Jenkins on Ubuntu 16.04.

Implement Container Logging and Monitoring

When working with containers, it's important to think about the logging infrastructure you will use to manage and store logs for all your running and stopped containers. There are multiple container-level patterns you can use for logging, and also multiple Kubernetes-level patterns.

In Kubernetes, by default containers use the json-file Docker logging driver, which captures the stdout and stderr streams and writes them to JSON files on the Node where the container is running. Sometimes logging directly to stderr and stdout may not be enough for your application container, and you may want to pair the app container with a logging sidecar container in a Kubernetes Pod. This sidecar container can then pick up logs from the filesystem, a local socket, or the systemd journal, granting you a little more flexibility than simply using the stderr and stdout streams. This container can also do some processing and then stream enriched logs to stdout/stderr, or directly to a logging backend. To learn more about Kubernetes logging patterns, consult the Kubernetes logging and monitoring section of this tutorial.

How your application logs at the container level will depend on its complexity. For simple, single-purpose microservices, logging directly to stdout/stderr and letting Kubernetes pick up these streams is the recommended approach, as you can then leverage the kubectl logs command to access log streams from your Kubernetes-deployed containers.

Similar to logging, you should begin thinking about monitoring in a container and cluster-based environment. Docker provides the helpful docker stats command for grabbing standard metrics like CPU and memory usage for running containers on the host, and exposes even more metrics through the Remote REST API. Additionally, the open-source tool cAdvisor (installed on Kubernetes Nodes by default) provides more advanced functionality like historical metric collection, metric data export, and a helpful web UI for sorting through the data.

However, in a multi-node, multi-container production environment, more complex metrics stacks like Prometheus and Grafana may help organize and monitor your containers' performance data.

Summary

In these sections, we briefly discussed some best practices for building containers, setting up a CI/CD pipeline and image registry, as well as some considerations for increasing observability into your containers.

To learn more about optimizing containers for Kubernetes, consult Building Optimized Containers for Kubernetes.
To learn more about CI/CD, consult An Introduction to Continuous Integration, Delivery, and Deployment and An Introduction to CI/CD Best Practices.

In the next section, we’ll explore Kubernetes features that allow you to run and scale your containerized app in a cluster.

Deploying on Kubernetes

At this point, you’ve containerized your app and implemented logic to maximize its portability and observability in Cloud Native environments. We’ll now explore Kubernetes features that provide simple interfaces for managing and scaling your apps in a Kubernetes cluster.

Write Deployment and Pod Configuration Files

Once you've containerized your application and published it to a registry, you can now deploy it into a Kubernetes cluster using the Pod workload. The smallest deployable unit in a Kubernetes cluster is not a container but a Pod. Pods typically consist of an application container (like a containerized Flask web app), or an app container and any “sidecar” containers that perform some helper function like monitoring or logging. Containers in a Pod share storage resources, a network namespace, and port space. They can communicate with each other using localhost and can share data using mounted volumes. Addtionally, the Pod workload allows you to define Init Containers that run setup scripts or utilities before the main app container begins running.

Pods are typically rolled out using Deployments, which are Controllers defined by YAML files that declare a particular desired state. For example, an application state could be running three replicas of the Flask web app container and exposing port 8080. Once created, the control plane gradually brings the actual state of the cluster to match the desired state declared in the Deployment by scheduling containers onto Nodes as required. To scale the number of application replicas running in the cluster, say from 3 up to 5, you update the replicas field of the Deployment configuration file, and then kubectl apply the new configuration file. Using these configuration files, scaling and deployment operations can all be tracked and versioned using your existing source control services and integrations.

Here’s a sample Kubernetes Deployment configuration file for a Flask app:

flask_deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
  labels:
    app: flask-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - name: flask
        image: sammy/flask_app:1.0
        ports:
        - containerPort: 8080

This Deployment launches 3 Pods that run a container called flask using the sammy/flask_app image (version 1.0) with port 8080 open. The Deployment is called flask-app.

To learn more about Kubernetes Pods and Deployments, consult the Pods and Deployments sections of the official Kubernetes documentation.

Configure Pod Storage

Kubernetes manages Pod storage using Volumes, Persistent Volumes (PVs) and Persistent Volume Claims (PVCs). Volumes are the Kubernetes abstraction used to manage Pod storage, and support most cloud provider block storage offerings, as well as local storage on the Nodes hosting the running Pods. To see a full list of supported Volume types, consult the Kubernetes documentation.

For example, if your Pod contains two NGINX containers that need to share data between them (say the first, called nginx serves web pages, and the second, called nginx-sync fetches the pages from an external location and updates the pages served by the nginx container), your Pod spec would look something like this (here we use the emptyDir Volume type):

pod_volume.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: nginx-web
      mountPath: /usr/share/nginx/html

  - name: nginx-sync
    image: nginx-sync
    volumeMounts:
    - name: nginx-web
      mountPath: /web-data

  volumes:
  - name: nginx-web
    emptyDir: {}

We use a volumeMount for each container, indicating that we'd like to mount the nginx-web volume containing the web page files at /usr/share/nginx/html in the nginx container and at /web-data in the nginx-sync container. We also define a volume called nginx-web of type emptyDir.

In a similar fashion, you can configure Pod storage using cloud block storage products by modifying the volume type from emptyDir to the relevant cloud storage volume type.

The lifecycle of a Volume is tied to the lifecycle of the Pod, but not to that of a container. If a container within a Pod dies, the Volume persists and the newly launched container will be able to mount the same Volume and access its data. When a Pod gets restarted or dies, so do its Volumes, although if the Volumes consist of cloud block storage, they will simply be unmounted with data still accessible by future Pods.

To preserve data across Pod restarts and updates, the PersistentVolume (PV) and PersistentVolumeClaim (PVC) objects must be used.

PersistentVolumes are abstractions representing pieces of persistent storage like cloud block storage volumes or NFS storage. They are created separately from PersistentVolumeClaims, which are demands for pieces of storage by developers. In their Pod configurations, developers request persistent storage using PVCs, which Kubernetes matches with available PV Volumes (if using cloud block storage, Kubernetes can dynamically create PersistentVolumes when PersistentVolumeClaims are created).

If your application requires one persistent volume per replica, which is the case with many databases, you should not use Deployments but use the StatefulSet controller, which is designed for apps that require stable network identifiers, stable persistent storage, and ordering guarantees. Deployments should be used for stateless applications, and if you define a PersistentVolumeClaim for use in a Deployment configuration, that PVC will be shared by all the Deployment's replicas.

To learn more about the StatefulSet controller, consult the Kubernetes documentation. To learn more about PersistentVolumes and PersistentVolume claims, consult the Kubernetes storage documentation.

Injecting Configuration Data with Kubernetes

Similar to Docker, Kubernetes provides the env and envFrom fields for setting environment variables in Pod configuration files. Here's a sample snippet from a Pod configuration file that sets the HOSTNAME environment variable in the running Pod to my_hostname :

sample_pod.yaml

...
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: HOSTNAME
          value: my_hostname
...

This allows you to move configuration out of Dockerfiles and into Pod and Deployment configuration files. A key advantage of further externalizing configuration from your Dockerfiles is that you can now modify these Kubernetes workload configurations (say, by changing the HOSTNAME value to my_hostname_2) separately from your application container definitions. Once you modify the Pod configuration file, you can then redeploy the Pod using its new environment, while the underlying container image (defined via its Dockerfile) does not need to be rebuilt, tested, and pushed to a repository. You can also version these Pod and Deployment configurations separately from your Dockerfiles, allowing you to quickly detect breaking changes and further separate config issues from application bugs.

Kubernetes provides another construct for further externalizing and managing configuration data: ConfigMaps and Secrets.

ConfigMaps and Secrets

ConfigMaps allow you to save configuration data as objects that you then reference in your Pod and Deployment configuration files, so that you can avoid hardcoding configuration data and reuse it across Pods and Deployments.

Here's an example, using the Pod config from above. We'll first save the HOSTNAME environment variable as a ConfigMap, and then reference it in the Pod config:

kubectl create configmap hostname --from-literal=HOSTNAME=my_host_name

To reference it from the Pod configuration file, we use the the valueFrom and configMapKeyRef constructs:

sample_pod_configmap.yaml

...
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: HOSTNAME
          valueFrom:
            configMapKeyRef:
              name: hostname
              key: HOSTNAME
...

So the HOSTNAME environment variable's value has been completely externalized from configuration files. We can then update these variables across all Deployments and Pods referencing them, and restart the Pods for the changes to take effect.

If your applications use configuration files, ConfigMaps additionally allow you to store these files as ConfigMap objects (using the --from-file flag), which you can then mount into containers as configuration files.

Secrets provide the same essential functionality as ConfigMaps, but should be used for sensitive data like database credentials as the values are base64-encoded.

To learn more about ConfigMaps and Secrets consult the Kubernetes documentation.

Create Services

Once you have your application up and running in Kubernetes, every Pod will be assigned an (internal) IP address, shared by its containers. If one of these Pods is removed or dies, newly started Pods will be assigned different IP addresses.

For long-running services that expose functionality to internal and/or external clients, you may wish to grant a set of Pods performing the same function (or Deployment) a stable IP address that load balances requests across its containers. You can do this using a Kubernetes Service.

Kubernetes Services have 4 types, specified by the type field in the Service configuration file:

ClusterIP: This is the default type, which grants the Service a stable internal IP accessible from anywhere inside of the cluster.
NodePort: This will expose your Service on each Node at a static port, between 30000-32767 by default. When a request hits a Node at its Node IP address and the NodePort for your service, the request will be load balanced and routed to the application containers for your service.
LoadBalancer: This will create a load balancer using your cloud provider's load balancing product, and configure a NodePort and ClusterIP for your Service to which external requests will be routed.
ExternalName: This Service type allows you to map a Kubernetes Service to a DNS record. It can be used for accessing external services from your Pods using Kubernetes DNS.

Note that creating a Service of type LoadBalancer for each Deployment running in your cluster will create a new cloud load balancer for each Service, which can become costly. To manage routing external requests to multiple services using a single load balancer, you can use an Ingress Controller. Ingress Controllers are beyond the scope of this article, but to learn more about them you can consult the Kubernetes documentation. A popular simple Ingress Controller is the NGINX Ingress Controller.

Here’s a simple Service configuration file for the Flask example used in the Pods and Deployments section of this guide:

flask_app_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: flask-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: flask-app
  type: LoadBalancer

Here we choose to expose the flask-app Deployment using this flask-svc Service. We create a cloud load balancer to route traffic from load balancer port 80 to exposed container port 8080.

To learn more about Kubernetes Services, consult the Services section of the Kubernetes docs.

Logging and Monitoring

Parsing through individual container and Pod logs using kubectl logs and docker logs can get tedious as the number of running applications grows. To help you debug application or cluster issues, you should implement centralized logging. At a high level, this consists of agents running on all the worker nodes that process Pod log files and streams, enrich them with metadata, and forward the logs off to a backend like Elasticsearch. From there, log data can be visualized, filtered, and organized using a visualization tool like Kibana.

In the container-level logging section, we discussed the recommended Kubernetes approach of having applications in containers log to the stdout/stderr streams. We also briefly discussed logging sidecar containers that can grant you more flexibility when logging from your application. You could also run logging agents directly in your Pods that capture local log data and forward them directly to your logging backend. Each approach has its pros and cons, and resource utilization tradeoffs (for example, running a logging agent container inside of each Pod can become resource-intensive and quickly overwhelm your logging backend). To learn more about different logging architectures and their tradeoffs, consult the Kubernetes documentation.

In a standard setup, each Node runs a logging agent like Filebeat or Fluentd that picks up container logs created by Kubernetes. Recall that Kubernetes creates JSON log files for containers on the Node (in most installations these can be found at /var/lib/docker/containers/). These should be rotated using a tool like logrotate. The Node logging agent should be run as a DaemonSet Controller, a type of Kubernetes Workload that ensures that every Node runs a copy of the DaemonSet Pod. In this case the Pod would contain the logging agent and its configuration, which processes logs from files and directories mounted into the logging DaemonSet Pod.

Similar to the bottleneck in using kubectl logs to debug container issues, eventually you may need to consider a more robust option than simply using kubectl top and the Kubernetes Dashboard to monitor Pod resource usage on your cluster. Cluster and application-level monitoring can be set up using the Prometheus monitoring system and time-series database, and Grafana metrics dashboard. Prometheus works using a "pull" model, which scrapes HTTP endpoints (like /metrics/cadvisor on the Nodes, or the /metrics application REST API endpoints) periodically for metric data, which it then processes and stores. This data can then be analyzed and visualized using Grafana dashboard. Prometheus and Grafana can be launched into a Kubernetes cluster like any other Deployment and Service.

For added resiliency, you may wish to run your logging and monitoring infrastructure on a separate Kubernetes cluster, or using external logging and metrics services.

Conclusion

Migrating and modernizing an application so that it can efficiently run in a Kubernetes cluster often involves non-trivial amounts of planning and architecting of software and infrastructure changes. Once implemented, these changes allow service owners to continuously deploy new versions of their apps and easily scale them as necessary, with minimal amounts of manual intervention. Steps like externalizing configuration from your app, setting up proper logging and metrics publishing, and configuring health checks allow you to fully take advantage of the Cloud Native paradigm that Kubernetes has been designed around. By building portable containers and managing them using Kubernetes objects like Deployments and Services, you can fully use your available compute infrastructure and development resources.

How to Install, Run, and Connect to Jupyter Notebook on a Remote Server

2018-09-11T22:38:20Z

The author selected the Apache Software Foundation to receive a $100 donation as part of the Write for DOnations program.

Introduction

Jupyter Notebook is an open-source, interactive web application that allows you to write and run computer code in more than 40 programming languages, including Python, R, Julia, and Scala. A product from Project Jupyter, Jupyter Notebook is useful for iterative coding as it allows you to write a small snippet of code, run it, and return the result.

Jupyter Notebook provides the ability to create notebook documents, referred to simply as "notebooks". Notebooks created from the Jupyter Notebook are shareable, reproducible research documents which include rich text elements, equations, code and their outputs (figures, tables, interactive plots). Notebooks can also be exported into raw code files, HTML or PDF documents, or used to create interactive slideshows or web pages.

This article will walk you through how to install and configure the Jupyter Notebook application on an Ubuntu 18.04 web server and how to connect to it from your local computer. Additionally, we will also go over how to use Jupyter Notebook to run some example Python code.

Prerequisites

To complete this tutorial, you will need:

One Ubuntu 18.04 server instance. This server must have a non-root user with sudo privileges and a firewall configured. Set this up by following our initial server setup guide.
Python 3, pip, and the Python venv module installed on the server. Do this by following Steps 1 and 2 of our tutorial on How To Install Python 3 and Set Up a Local Programming Environment on Ubuntu 18.04.
A modern web browser running on your local computer which you will use to access Jupyter Notebook.

Additionally, if your local computer is running Windows, you will need to install PuTTY on it in order to establish an SSH tunnel to your server. Follow our guide on How to Create SSH Keys with PuTTY on Windows to download and install PuTTY.

Step 1 — Installing Jupyter Notebook

Since notebooks are used to write, run and see the result of small snippets of code, you will first need to set up the programming language support. Jupyter Notebook uses a language-specific kernel, a computer program that runs and introspects code. Jupyter Notebook has many kernels in different languages, the default being IPython. In this tutorial, you will set up Jupyter Notebook to run Python code through the IPython kernel.

Assuming that you followed the tutorials linked in the Prerequisites section, you should have Python 3, pip and a virtual environment installed. The examples in this guide follow the convention used in the prerequisite tutorial on installing Python 3, which names the virtual environment "my_env", but you should feel free to rename it.

Begin by activating the virtual environment:

source my_env/bin/activate

Following this, your prompt will be prefixed with the name of your environment.

Now that you're in your virtual environment, go ahead and install Jupyter Notebook:

python3 -m pip install jupyter

If the installation was successful, you will see an output similar to the following:

Output. . .
Successfully installed MarkupSafe-1.0 Send2Trash-1.5.0 backcall-0.1.0 bleach-2.1.3 decorator-4.3.0 entrypoints-0.2.3 html5lib-1.0.1 ipykernel-4.8.2 ipython-6.4.0 ipython-genutils-0.2.0 ipywidgets-7.2.1 jedi-0.12.0 jinja2-2.10 jsonschema-2.6.0 jupyter-1.0.0 jupyter-client-5.2.3 jupyter-console-5.2.0 jupyter-core-4.4.0 mistune-0.8.3 nbconvert-5.3.1 nbformat-4.4.0 notebook-5.5.0 pandocfilters-1.4.2 parso-0.2.0 pexpect-4.5.0 pickleshare-0.7.4 prompt-toolkit-1.0.15 ptyprocess-0.5.2 pygments-2.2.0 python-dateutil-2.7.3 pyzmq-17.0.0 qtconsole-4.3.1 simplegeneric-0.8.1 six-1.11.0 terminado-0.8.1 testpath-0.3.1 tornado-5.0.2

With that, Jupyter Notebook has been installed onto your server. Next, we will go over how to run the application.

Step 2 — Running the Jupyter Notebook

Jupyter Notebook must be run from your VPS so that you can connect to it from your local machine using an SSH Tunnel and your favorite web browser.

To run the Jupyter Notebook server, enter the following command:

jupyter notebook

After running this command, you will see output similar to the following:

Output
[I 19:46:22.031 NotebookApp] Writing notebook server cookie secret to /home/sammy/.local/share/jupyter/runtime/notebook_cookie_secret
[I 19:46:22.365 NotebookApp] Serving notebooks from local directory: /home/sammy/environments
[I 19:46:22.365 NotebookApp] 0 active kernels
[I 19:46:22.366 NotebookApp] The Jupyter Notebook is running at:
[I 19:46:22.366 NotebookApp] http://localhost:8888/?token=Example_Jupyter_Token_3cadb8b8b7005d9a46ca4d6675
[I 19:46:22.366 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 19:46:22.366 NotebookApp] No web browser found: could not locate runnable browser.
[C 19:46:22.367 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=Example_Jupyter_Token_3cadb8b8b7005d9a46ca4d6675&tokenExample_Jupyter_Token_3cadb8b8b7005d9a46ca4d6675

You might notice in the output that there is a No web browser found warning. This is to be expected, since the application is running on a server and you likely haven't installed a web browser onto it. This guide will go over how to connect to the Notebook on the server using SSH tunneling in the next section.

For now, exit the Jupyter Notebook by pressing CTRL+C followed by y, and then pressing ENTER to confirm:

Output
Shutdown this notebook server (y/[n])? y
[C 20:05:47.654 NotebookApp] Shutdown confirmed
[I 20:05:47.654 NotebookApp] Shutting down 0 kernels

Then log out of the server by using the exit command:

exit

You've just run Jupyter Notebook on your server. However, in order to access the application and start working with notebooks, you'll need to connect to the application using SSH tunneling and a web browser on your local computer.

Step 3 — Connecting to the Jupyter Notebook Application with SSH Tunneling

SSH tunneling is a simple and fast way to connect to the Jupyter Notebook application running on your server. Secure shell (more commonly known as SSH) is a network protocol which enables you to connect to a remote server securely over an unsecured network.

The SSH protocol includes a port forwarding mechanism that allows you to tunnel certain applications running on a specific port number on a server to a specific port number on your local computer. We will learn how to securely "forward" the Jupyter Notebook application running on your server (on port 8888, by default) to a port on your local computer.

The method you use for establishing an SSH tunnel will depend on your local computer's operating system. Jump to the subsection below that is most relevant for your machine.

Note: It's possible to set up and install the Jupyter Notebook using the DigitalOcean Web Console, but connecting to the application via an SSH tunnel must be done through the terminal or with PuTTY.

SSH Tunneling using macOS or Linux

If your local computer is running Linux or macOS, it's possible to establish an SSH tunnel just by running a single command.

ssh is the standard command to open an SSH connection, but when used with the -L directive, you can specify that a given port on the local host (that is, your local machine) will be forwarded to a given host and port on the remote host (in this case, your server). This means that whatever is running on the specified port on the remote server (8888, Jupyter Notebook's default port) will appear on the specified port on your local computer (8000 in the example command).

To establish your own SSH tunnel, run the following command. Feel free to change port 8000 to one of your choosing if, for example, 8000 is in use by another process. It is recommended that you use a port greater than or equal to 8000, as those port numbers are unlikely to be used by another process. Be sure to include your own server's IP address and the name of your server's non-root user:

ssh -L 8000:localhost:8888 sammy@your_server_ip

If there are no errors from this command, it will log you into your remote server. From there, activate the virtual environment:

source ~/environments/my_env/bin/activate

Then run the Jupyter Notebook application:

jupyter notebook

To connect to Jupyter Notebook, use your favorite web browser to navigate to the local port on the local host: http://localhost:8000. Now that you're connected to Jupyter Notebook, continue on to Step 4 to learn how to use it.

SSH Tunneling using Windows and PuTTY

PuTTY is an open-source SSH client for Windows which can be used to connect to your server. After downloading and installing PuTTY on your Windows machine (as described in the prerequisite tutorial), open the program and enter your server URL or IP address, as shown here:

Next, click + SSH at the bottom of the left pane, and then click Tunnels. In this window, enter the port that you want to use to access Jupyter on your local machine (8000 ). It is recommended to use a port greater or equal to 8000 as those port numbers are unlikely to be used by another process. If 8000 is used by another process, though, select a different, unused port number. Next, set the destination as localhost:8888, since port 8888 is the one that Jupyter Notebook is running on. Then click the Add button and the ports should appear in the Forwarded ports field:

Finally, click the Open button. This will both connect your machine to the server via SSH and tunnel the desired ports. If no errors show up, go ahead and activate your virtual environment:

source ~/environments/my_env/bin/activate

Then run Jupyter Notebook:

jupyter notebook

Next, navigate to the local port in your favorite web browser, for example http://localhost:8000 (or whatever port number you chose), to connect to the Jupyter Notebook instance running on the server. Now that you're connected to Jupyter Notebook, continue on to Step 4 to learn how to use it.

Step 4 — Using Jupyter Notebook

When accessed through a web browser, Jupyter Notebook provides a Notebook Dashboard which acts as a file browser and gives you an interface for creating, editing and exploring notebooks. Think of these notebooks as documents (saved with a .ipynb file extension) which you populate with any number of individual cells. Each cell holds an interactive text editor which can be used to run code or write rendered text. Additionally, notebooks allow you to write and run equations, include other rich media, such as images or interactive plots, and they can be exported and shared in various formats (.ipyb, .pdf, .py). To illustrate some of these functions, we'll create a notebook file from the Notebook Dashboard, write a simple text board with an equation, and run some basic Python 3 code.

By this point you should have connected to the server using an SSH tunnel and started the Jupyter Notebook application from your server. After navigating to http://localhost:8000, you will be presented with a login page:

In the Password or token field at the top, enter the token shown in the output after you ran jupyter notebook from your server:

Output[I 20:35:17.004 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret
[I 20:35:17.314 NotebookApp] Serving notebooks from local directory: /home/sammy
[I 20:35:17.314 NotebookApp] 0 active kernels
[I 20:35:17.315 NotebookApp] The Jupyter Notebook is running at:
[I 20:35:17.315 NotebookApp] http://localhost:8888/?token=Example_Jupyter_Token_3cadb8b8b7005d9a46ca4d6675
[I 20:35:17.315 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 20:35:17.315 NotebookApp] No web browser found: could not locate runnable browser.
[C 20:35:17.316 NotebookApp]
. . .

Alternatively, you can copy that URL from your terminal output and paste it into your browser's address bar.

Automatically, Jupyter notebook will show all of the files and folders stored in the directory from which it's run. Create a new notebook file by clicking New then Python 3 at the top-right of the Notebook Dashboard:

Within this new notebook, change the first cell to accept markdown syntax by clicking Cell > Cell Type > Markdown on the navigation bar at the top. In addition to markdown, this Cell Type also allows you to write equations in LaTeX. For example, type the following into the cell after changing it to markdown:

# Simple Equation

Let us now implement the following equation in Python:
$$ y = x^2$$

where $x = 2$

To turn the markdown into rich text, press CTRL + ENTER and the following should be the result:

You can use the markdown cells to make notes and document your code.

Now, let's implement a simple equation and print the result. Click Insert > Insert Cell Below to insert a cell. In this new cell, enter the following code:

x = 2
y = x*x
print(y)

To run the code, press CTRL + ENTER, and the following will be the result:

These are some relatively simple examples of what you can do with Jupyter Notebook. However, it is a very powerful application with many potential use cases. From here, you can add some Python libraries and use the notebook as you would with any other Python development environment.

Conclusion

You should be now able to write reproducible Python code and text using the Jupyter Notebook running on a remote server. To get a quick tour of Jupyter Notebook, click Help in the top navigation bar and select User Interface Tour as shown here:

If you're interested, we encourage you to learn more about Jupyter Notebook by going through the Project Jupyter documentation. Additionally, you can build on what you learned in this tutorial by learning how to code in Python 3.

How To Install R on Debian 9

2018-09-11T17:52:58Z

Introduction

R is an open-source programming language that specializes in statistical computing and graphics. Supported by the R Foundation for Statistical Computing, it is widely used for developing statistical software and performing data analysis. An increasingly popular and extensible language with an active community, R offers many user-generated packages for specific areas of study, which makes it applicable to many fields.

In this tutorial, we will install R and show how to add packages from the official Comprehensive R Archive Network (CRAN).

Prerequisites

To follow along with this tutorial, you will need a Debian 9 server with:

at least 1GB of RAM
a non-root user with sudo privileges

To learn how to achieve this setup, follow our Debian 9 initial server setup guide.

Once these prerequisites are in place, you’re ready to begin.

Step 1 — Installing Dependencies

Because R is a fast-moving project, the latest stable version isn’t always available from Debian’s repositories, so we’ll need to add the external repository maintained by CRAN. In order to do this, we’ll need to install some dependencies for the Debian 9 cloud image.

To perform network operations that manage and download certificates, we need to install dirmngr so that we can add the external repository.

sudo apt install dirmngr --install-recommends

To add a PPA reference to Debian, we’ll need to use the add-apt-repository command. For installations where this command may not available, you can add this utility to your system by installing software-properties-common:

sudo apt install software-properties-common

Finally, to ensure that we have HTTPS support for secure protocols, we’ll install the following tool:

sudo apt install apt-transport-https

With these dependencies in place, we’re ready to install R.

Step 2 — Installing R

For the most recent version of R, we’ll be installing from the CRAN repositories.

Note: CRAN maintains the repositories within their network, but not all external repositories are reliable. Be sure to install only from trusted sources.

Let’s first add the relevant GPG key.

sudo apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF'

When we run the command, we’ll receive the following output:

OutputExecuting: /tmp/apt-key-gpghome.k3UoM7WQGq/gpg.1.sh --keyserver keys.gnupg.net --recv-key E19F5F87128899B192B1A2C2AD5F960A256A04AF
gpg: key AD5F960A256A04AF: public key "Johannes Ranke (Wissenschaftlicher Berater) <johannes.ranke@jrwb.de>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Once we have the trusted key, we can add the repository. Note that if you’re not using Debian 9 (Stretch), you can look at the supported R Project Debian branches, named for each release.

sudo add-apt-repository 'deb https://cloud.r-project.org/bin/linux/debian stretch-cran35/'

Now, we’ll need to run update after this in order to include package manifests from the new repository.

sudo apt update

Among the output should be a line similar to the following:

Among the output that displays, you should identify lines similar to the following:

Output...
Get:6 https://cloud.r-project.org/bin/linux/debian stretch-cran35/ InRelease [4,371 B]
Get:7 https://cloud.r-project.org/bin/linux/debian stretch-cran35/ Packages [50.1 kB]
...

If the lines above appear in the output from the update command, we’ve successfully added the repository. We can be sure we won’t accidentally install an older version.

At this point, we’re ready to install R with the following command.

sudo apt install r-base

If prompted to confirm installation, press y to continue.

As of the time of writing, the latest stable version of R from CRAN is 3.5.1, which is displayed when you start R.

Since we’re planning to install an example package for every user on the system, we’ll start R as root so that the libraries will be available to all users automatically. Alternatively, if you run the R command without sudo, a personal library can be set up for your user.

sudo -i R

Output
R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
...
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

This confirms that we’ve successfully installed R and entered its interactive shell.

Step 3 — Installing R Packages from CRAN

Part of R’s strength is its available abundance of add-on packages. For demonstration purposes, we’ll install txtplot, a library that outputs ASCII graphs that include scatterplot, line plot, density plot, acf and bar charts:

install.packages('txtplot')

Note: The following output shows where the package will be installed.

Output...
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
. . .

This site-wide path is available because we ran R as root. This is the correct location to make the package available to all users.

When the installation is complete, we can load txtplot:

library('txtplot')

If there are no error messages, the library has successfully loaded. Let’s put it in action now with an example which demonstrates a basic plotting function with axis labels. The example data, supplied by R's datasets package, contains the speed of cars and the distance required to stop based on data from the 1920s:

txtplot(cars[,1], cars[,2], xlab = 'speed', ylab = 'distance')

Output      +----+-----------+------------+-----------+-----------+--+
  120 +                                                   *    +
      |                                                        |
d 100 +                                                   *    +
i     |                                    *                *  |
s  80 +                          *         *                   +
t     |                                       * *    *    *    |
a  60 +                          *  *      *    *      *       +
n     |                        *         * *  * *              |
c  40 +                *       * *    *  *    * *              +
e     |         *      *  * *  * *  *                          |
   20 +           *    *  * *       *                          +
      |  *      *    *                                         |
    0 +----+-----------+------------+-----------+-----------+--+
           5          10           15          20          25   
                                speed

If you are interested to learn more about txtplot, use help(txtplot) from within the R interpreter.

Any precompiled package can be installed from CRAN with install.packages(). To learn more about what’s available, you can find a listing of official packages organized by name via the Available CRAN Packages By Name list.

To exit R, you can type q(). Unless you want to save the workspace image, you can press n.

Conclusion

With R successfully installed on your server, you may be interested in this guide on installing the RStudio Server to bring an IDE to the server-based deployment you just completed. You can also learn how to set up a Shiny server to convert your R code into interactive web pages.

For more information on how to install R packages by leveraging different tools, you can read about how to install directly from GitHub, BitBucket or other locations. This will allow you to take advantage of the very latest work from the active community.

Como Criar um Cluster Kubernetes 1.10 Usando Kubeadm no CentOS 7

2018-09-11T00:53:39Z

O autor escolheu o Free and Open Source Fund para receber uma doação como parte do programa Write for DOnations.

Introdução

O Kubernetes é um sistema de orquestração de container em escala. Inicialmente desenvolvido pelo Google baseado em suas experiências executando containers em produção. O Kubernetes é open source e desenvolvido ativamente por uma comunidade em todo o mundo.

O Kubeadm atomatiza a instalação e a configuração de componentes do Kubernetes tais como o servidor de API, o Controller Manager, e o Kube DNS. Contudo, ele não cria usuários ou lida com a instalação de dependências no nível do sistema operacional e sua configuração. Para essa tarefas preliminares, é possível utilizar uma ferramenta de gerência de configuração como o Ansible ou o SaltStack. A utilização dessas ferramentas torna a criação de clusters adicionais ou a recriação de clusters existentes muito mais simples e menos propensa a erros.

Neste guia, você vai configurar um cluster Kubernetes a partir do zero utilizando o Ansible e o Kubeadm, e a seguir fazer o deploy de uma aplicação Nginx containerizada nele.

Objetivos

Seu cluster irá incluir os seguintes recursos físicos:

Um node master

O node master (um node no Kubernetes refere-se a um servidor) é responsável por gerenciar o estado do cluster. Ele roda o Etcd, que armazena dados de cluster entre componentes que fazem o scheduling de cargas de trabalho para nodes de trabalho.

Dois nodes worker

Nodes worker são os servidores onde suas cargas de trabalho (i.e. aplicações e serviços containerizados) irão executar. Um worker continuará a executar sua carga de trabalho uma vez que estejam atribuídos a ela, mesmo se o master for desativado quando o scheduling estiver concluído. A capacidade de um cluster pode ser aumentada adicionando workers.

Após a conclusão desse guia, você terá um cluster pronto para executar aplicações containerizadas, desde que os servidores no cluster tenham recursos suficientes de CPU e RAM para suas aplicações consumirem. Quase todas as aplicações Unix tradicionais, incluindo aplicações web, bancos de dados, daemons, e ferramentas de linha de comando podem ser containerizadas e feitas para rodar no cluster. O cluster em si consumirá cerca de 300-500MB de memória e 10% de CPU em cada node.

Uma vez que o cluster esteja configurado, você fará o deploy do servidor web Nginx nele para assegurar que ele está executando as cargas de trabalho corretamente.

Pré-requisitos

Um par de chaves SSH em sua máquina Linux/macOS/BSD local. Se você não tiver usado chaves SSH antes, você pode aprender como configurá-las seguindo esta explicação em como configurar chaves SSH em sua máquina local.
Três servidores rodando CentOS 7 com pelo menos 1GB de RAM. Você deve ser capaz de fazer SSH em cada servidor como usuário root com o seu par de chaves SSH. Certifique-se também de adicionar sua chave pública à conta do usuário do centos no node master. Se você precisar de orientação sobre como adicionar uma chave SSH a uma conta de usuário específica, consulte este tutorial sobre Como Configurar Chaves SSH no CentOS7.
Ansible instalado em sua máquina local. Para instruções de instalação, siga a documentação oficial da instalação do Ansible.
Familiaridade com o playbooks do Ansible. Para uma revisão, verifique Configuration Management 101: Writing Ansible Playbooks.
Conhecimento de como lançar um container a partir de uma imagem Docker. Dê uma olhada no "Passo 5 — Executando um Container Docker" em Como instalar e usar o Docker no CentOS 7 se precisar relembrar.

Passo 1 — Configurando o Diretório da Área de Trabalho e o Arquivo de Inventário Ansible

Nessa seção, você vai criar um diretório em sua máquina local que irá servir como sua área de trabalho. Você configurará o Ansible localmente para que ele possa se comunicar e executar comandos em seus servidores remotos. Depois disso pronto, você irá criar um arquivo hosts contendo informações de inventário tais como os endereços IP de seus servidores e os grupos aos quais cada servidor pertence.

Dos seus três servidores, um será o master com um IP exibido como master_ip. Os outros dois servidores serão workers e terão os IPs worker_1_ip e worker_2_ip.

Crie um diretório chamado ~/kube-cluster no diretório home de sua máquina local e faça um cd para dentro dele:

mkdir ~/kube-cluster
cd ~/kube-cluster

Esse diretório será sua área de trabalho para o restante desse tutorial e conterá todos os seus playbooks de Ansible. Ele também será o diretório no qual você irá executar todos os comandos locais.

Crie um arquivo chamado ~/kube-cluster/hosts usando o vi ou o seu editor de textos favorito:

vi ~/kube-cluster/hosts

Pressione i para inserir o seguinte texto ao arquivo, que irá especificar informações sobre a estrutura lógica do cluster:

~/kube-cluster/hosts


[masters]
master ansible_host=master_ip ansible_user=root

[workers]
worker1 ansible_host=worker_1_ip ansible_user=root
worker2 ansible_host=worker_2_ip ansible_user=root

Quando tiver terminado, pressione ESC seguido de :wq para gravar as alterações no arquvo e sair.

Você deve se lembrar de que arquivos de inventário no Ansible são utilizados para especificar informações de servidor tais como endereços IP, usuários remotos, e agrupamentos de servidores para tratar como uma unidade única para a execução de comandos. O ~/kube-cluster/hosts será o seu arquivo de inventário e você adicionou dois grupos Ansible a ele (masters e workers) especificando a estrutura lógica do seu cluster.

No grupo masters, existe uma entrada de servidor chamada "master" que lista o IP do node master (master_ip) e especifica que o Ansible deve executar comandos remotos como root.

De maneira similar, no grupo workers, existem duas entradas para os servidores workers (worker_1_ip e worker_2_ip) que também especificam o ansible_user como root.

Tendo configurado o inventário do servidor com grupos, vamos passar a instalar dependências no nível do sistema operacional e a criar definições de configuração.

Passo 2 — Criando um Usuário Não-Root em Todos os Servidores Remotos

Nesta seção você irá criar um usuário não-root com privilégios sudo em todos os servidores para que você possa fazer SSH manualmente neles como um usuário sem privilégios. Isso pode ser útil se, por exemplo, você gostaria de ver informações do sistema com comandos como top/htop, ver a lista de containers em execução, ou alterar arquivos de configuração de propriedade do root. Estas operações são rotineiramente executadas durante a manutenção de um cluster, e a utilização de um usuário que não seja root para tarefas desse tipo minimiza o risco de modificação ou exclusão de arquivos importantes ou a realização não intencional de operações perigosas.

Crie um arquivo chamado ~/kube-cluster/initial.yml na área de trabalho:

vi ~/kube-cluster/initial.yml

A seguir, adicione o seguinte play ao arquivo para criar um usuário não-root com privilégios sudo em todos os servidores. Um play no Ansible é uma coleção de passos a serem realizados que visam servidores e grupos específicos. O seguinte play irá criar um usuário sudo não-root:

~/kube-cluster/initial.yml


- hosts: all
  become: yes
  tasks:
    - name: create the 'centos' user
      user: name=centos append=yes state=present createhome=yes shell=/bin/bash

    - name: allow 'centos' to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        line: 'centos ALL=(ALL) NOPASSWD: ALL'
        validate: 'visudo -cf %s'

    - name: set up authorized keys for the centos user
      authorized_key: user=centos key="{{item}}"
      with_file:
        - ~/.ssh/id_rsa.pub

Aqui está um detalhamento do que este playbook faz:

Cria um usuário não-root centos.
Configura o arquivo sudoers para permitir o usuário centos executar comandos sudo sem uma solicitação de senha.
Adiciona a chave pública em sua máquina local (normalmente ~/.ssh/id_rsa.pub) para a lista de chaves autorizadas do usuário remoto centos. Isto o permitirá fazer SSH para dentro de cada servidor como usuário centos.

Salve e feche o arquivo depois que tiver adicionado o texto.

Em seguida, rode o playbook localmente executando:

ansible-playbook -i hosts ~/kube-cluster/initial.yml

O comando será concluído dentro de dois a cinco minutos. Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [all] ****

TASK [Gathering Facts] ****
ok: [master]
ok: [worker1]
ok: [worker2]

TASK [create the 'centos' user] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [allow 'centos' user to have passwordless sudo] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [set up authorized keys for the centos user] ****
changed: [worker1] => (item=ssh-rsa AAAAB3...)
changed: [worker2] => (item=ssh-rsa AAAAB3...)
changed: [master] => (item=ssh-rsa AAAAB3...)

PLAY RECAP ****
master                     : ok=5    changed=4    unreachable=0    failed=0   
worker1                    : ok=5    changed=4    unreachable=0    failed=0   
worker2                    : ok=5    changed=4    unreachable=0    failed=0

Agora que a configuração preliminar está completa, você pode passar para a instalação de dependências específicas do Kubernetes.

Passo 3 — Instalando as Dependências do Kubernetes

Nesta seção, você irá instalar os pacotes no nível do sistema operacional necessários pelo Kubernetes com o gerenciador de pacotes yum do CentOS. Esses pacotes são:

Docker - um runtime de container. Este é o componente que executa seus containers. Suporte a outros runtimes como o rkt está em desenvolvimento ativo no Kubernetes.
kubeadm - uma ferramenta CLI que irá instalar e configurar os vários componentes de um cluster de uma maneira padrão.
kubelet - um serviço/programa de sistema que roda em todos os nodes e lida com operações no nível do node.
kubectl - uma ferramenta CLI usada para emitir comandos para o cluster através de seu servidor de API.

Crie um arquivo chamado ~/kube-cluster/kube-dependencies.yml na área de trabalho:

vi ~/kube-cluster/kube-dependencies.yml

Adicione os seguintes plays ao arquivo para instalar esses pacotes em seus servidores:

~/kube-cluster/kube-dependencies.yml


- hosts: all
  become: yes
  tasks:
   - name: install Docker
     yum:
       name: docker
       state: present
       update_cache: true

   - name: start Docker
     service:
       name: docker
       state: started

   - name: disable SELinux
     command: setenforce 0

   - name: disable SELinux on reboot
     selinux:
       state: disabled

   - name: ensure net.bridge.bridge-nf-call-ip6tables is set to 1
     sysctl:
      name: net.bridge.bridge-nf-call-ip6tables
      value: 1
      state: present

   - name: ensure net.bridge.bridge-nf-call-iptables is set to 1
     sysctl:
      name: net.bridge.bridge-nf-call-iptables
      value: 1
      state: present

   - name: add Kubernetes' YUM repository
     yum_repository:
      name: Kubernetes
      description: Kubernetes YUM repository
      baseurl: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
      gpgkey: https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
      gpgcheck: yes

   - name: install kubelet
     yum:
        name: kubelet
        state: present
        update_cache: true

   - name: install kubeadm
     yum:
        name: kubeadm
        state: present

   - name: start kubelet
     service:
       name: kubelet
       enabled: yes
       state: started

- hosts: master
  become: yes
  tasks:
   - name: install kubectl
     yum:
        name: kubectl
        state: present

O primeiro play no playbook faz o seguinte:

Instala o Docker, o runtime de container.
Inicia o serviço do Docker.
Desativa o SELinux, uma vez que ele ainda não é totalmente suportado pelo Kubernetes.
Define alguns valores sysctl relacionados ao netfilter, necessários para o trabalho em rede. Isso permitirá que o Kubernetes defina regras de iptables para receber tráfego de rede IPv4 e IPv6 em bridge nos nodes.
Adiciona o repositório YUM do Kubernetes às listas de repositórios de seus servidores remotos.
Instala kubelet e kubeadm.

O segundo play consiste de uma única tarefa que instala o kubectl no seu node master.

Salve e feche o arquivo quando você tiver terminado.

A seguir, execute o playbook:

ansible-playbook -i hosts ~/kube-cluster/kube-dependencies.yml

Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [all] ****

TASK [Gathering Facts] ****
ok: [worker1]
ok: [worker2]
ok: [master]

TASK [install Docker] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [disable SELinux] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [disable SELinux on reboot] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [ensure net.bridge.bridge-nf-call-ip6tables is set to 1] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [ensure net.bridge.bridge-nf-call-iptables is set to 1] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [start Docker] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [add Kubernetes' YUM repository] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [install kubelet] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [install kubeadm] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [start kubelet] ****
changed: [master]
changed: [worker1]
changed: [worker2]

PLAY [master] *****

TASK [Gathering Facts] *****
ok: [master]

TASK [install kubectl] ******
ok: [master]

PLAY RECAP ****
master                     : ok=9    changed=5    unreachable=0    failed=0   
worker1                    : ok=7    changed=5    unreachable=0    failed=0  
worker2                    : ok=7    changed=5    unreachable=0    failed=0

Após a execução, o Docker, o kubeadm e o kubelet estarão instalados em todos os seus servidores remotos. O kubectl não é um componente obrigatório e somente é necessário para a execução de comandos de cluster. A instalação dele somente no node master faz sentido nesse contexto, uma vez que você irá executar comandos kubectl somente a partir do master. Contudo, observe que os comandos kubectl podem ser executados a partir de quaisquer nodes worker ou a partir de qualquer máquina onde ele possa ser instalado e configurado para apontar para um cluster.

Todas as dependências de sistema agora estão instaladas. Vamos configurar o node master e inicializar o cluster.

Passo 4 — Configurando o Node Master

Nesta seção, você irá configurar o node master. Antes da criação de quaisquer playbooks, contudo, vale a pena cobrir alguns conceitos como Pods e Plugins de Rede do Pod, uma vez que seu cluster incluirá ambos.

Um pod é uma unidade atômica que executa um ou mais containers. Esses containers compartilham recursos tais como volumes de arquivo e interfaces de rede em comum. Os pods são a unidade básica de scheduling no Kubernetes: todos os containers em um pod têm a garantia de serem executados no mesmo node no qual foi feito o scheduling do pod.

Cada pod tem seu próprio endereço IP, e um pod em um node deve ser capaz de acessar um pod em outro node utilizando o IP do pod. Os containers em um único node podem se comunicar facilmente através de uma interface local. Contudo, a comunicação entre pods é mais complicada e requer um componente de rede separado que possa encaminhar o tráfego de maneira transparente de um pod em um node para um pod em outro node.

Essa funcionalidade é fornecida pelos plugins de rede para pods. Para este cluster vamos utilizar o Flannel, uma opção estável e de bom desempenho.

Crie um playbook Ansible chamado master.yml em sua máquina local:

vi ~/kube-cluster/master.yml

Adicione o seguinte play ao arquivo para inicializar o cluster e instalar o Flannel:

~/kube-cluster/master.yml


- hosts: master
  become: yes
  tasks:
    - name: initialize the cluster
      shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt
      args:
        chdir: $HOME
        creates: cluster_initialized.txt

    - name: create .kube directory
      become: yes
      become_user: centos
      file:
        path: $HOME/.kube
        state: directory
        mode: 0755

    - name: copy admin.conf to user's kube config
      copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/centos/.kube/config
        remote_src: yes
        owner: centos

    - name: install Pod network
      become: yes
      become_user: centos
      shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml >> pod_network_setup.txt
      args:
        chdir: $HOME
        creates: pod_network_setup.txt

Aqui está um detalhamento deste play:

A primeira tarefa inicializa o cluster executando kubeadm init. A passagem do argumento --pod-network-cidr=10.244.0.0/16 especifica a sub-rede privada que os IPs do pod serão atribuídos. O Flannel utiliza a sub-rede acima por padrão; estamos dizendo ao kubeadm para utilizar a mesma sub-rede.
A segunda tarefa cria um diretório .kube em /home/centos. Este diretório irá manter as informações de configuração tais como os arquivos de chaves do admin, que são requeridas para conectar no cluster, e o endereço da API do cluster.
A terceira tarefa copia o arquivo /etc/kubernetes/admin.conf que foi gerado a partir do kubeadm init para o diretório home do seu usuário não-root centos. Isso irá permitir que você utilize o kubectl para acessar o cluster recém-criado.
A última tarefa executa kubectl apply para instalar o Flannel. kubectl apply -f descriptor.[yml|json] é a sintaxe para dizer ao kubectl para criar os objetos descritos no arquivo descriptor.[yml|json]. O arquivo kube-flannel.yml contém as descrições dos objetos requeridos para a configuração do Flannel no cluster.

Salve e feche o arquivo quando você tiver terminado.

Execute o playbook:

ansible-playbook -i hosts ~/kube-cluster/master.yml

Na conclusão, você verá uma saída semelhante à seguinte:

Output
PLAY [master] ****

TASK [Gathering Facts] ****
ok: [master]

TASK [initialize the cluster] ****
changed: [master]

TASK [create .kube directory] ****
changed: [master]

TASK [copy admin.conf to user's kube config] *****
changed: [master]

TASK [install Pod network] *****
changed: [master]

PLAY RECAP ****
master                     : ok=5    changed=4    unreachable=0    failed=0

Para verificar o status do node master, faça SSH nele com o seguinte comando:

ssh centos@master_ip

Uma vez dentro do node master, execute:

kubectl get nodes

Agora você verá a seguinte saída:

OutputNAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    1d        v1.10.1

A saída informa que o node master concluiu todas as tarefas de inicialização e está em um estado Ready do qual pode começar a aceitar nodes worker e executar tarefas enviadas ao Servidor de API. Agora você pode adicionar os workers a partir de sua máquina local.

Passo 5 — Configurando os Nodes Worker

A adição de workers ao cluster envolve a execução de um único comando em cada um. Este comando inclui as informações necessárias sobre o cluster, tais como o endereço IP e a porta do Servidor de API do master, e um token seguro. Somentes os nodes que passam no token seguro estarão aptos a ingressar no cluster.

Navegue de volta para a sua área de trabalho e crie um playbook chamado workers.yml:

vi ~/kube-cluster/workers.yml

Adicione o seguinte texto ao arquivo para adicionar os workers ao cluster:

~/kube-cluster/workers.yml


- hosts: master
  become: yes
  gather_facts: false
  tasks:
    - name: get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

    - name: set join command
      set_fact:
        join_command: "{{ join_command_raw.stdout_lines[0] }}"


- hosts: workers
  become: yes
  tasks:
    - name: join cluster
      shell: "{{ hostvars['master'].join_command }} >> node_joined.txt"
      args:
        chdir: $HOME
        creates: node_joined.txt

Aqui está o que o playbook faz:

O primeiro play obtém o comando de junção que precisa ser executado nos nodes workers. Este comando estará no seguinte formato: kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>. Assim que obtiver o comando real com os valores apropriados de token e hash, a tarefa define isso como um fact para que o próximo play possa acessar essa informação.
O segundo play tem uma única tarefa que executa o comando de junção em todos os nodes worker. Na conclusão desta tarefa, os dois nodes worker farão parte do cluster.

Salve e feche o arquivo quando você tiver terminado.

Execute o playbook:

ansible-playbook -i hosts ~/kube-cluster/workers.yml

Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [master] ****

TASK [get join command] ****
changed: [master]

TASK [set join command] *****
ok: [master]

PLAY [workers] *****

TASK [Gathering Facts] *****
ok: [worker1]
ok: [worker2]

TASK [join cluster] *****
changed: [worker1]
changed: [worker2]

PLAY RECAP *****
master                     : ok=2    changed=1    unreachable=0    failed=0   
worker1                    : ok=2    changed=1    unreachable=0    failed=0  
worker2                    : ok=2    changed=1    unreachable=0    failed=0

Com a adição dos nodes worker, seu cluster está agora totalmente configurado e funcional, com os workers prontos para executar cargas de trabalho. Antes de fazer o scheduling de aplicações, vamos verificar se o cluster está funcionando conforme o esperado.

Step 6 — Verificando o Cluster

Às vezes, um cluster pode falhar durante a configuração porque um node está inativo ou a conectividade de rede entre o master e o worker não está funcionando corretamente. Vamos verificar o cluster e garantir que os nodes estejam operando corretamente.

Você precisará verificar o estado atual do cluster a partir do node master para garantir que os nodes estejam prontos. Se você se desconectou do node master, pode voltar e fazer SSH com o seguinte comando:

ssh centos@master_ip

Em seguida, execute o seguinte comando para obter o status do cluster:

kubectl get nodes

Você verá uma saída semelhante à seguinte:

OutputNAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    1d        v1.10.1
worker1   Ready     <none>    1d        v1.10.1 
worker2   Ready     <none>    1d        v1.10.1

Se todos os seus nodes têm o valor Ready para o STATUS, significa que eles são parte do cluster e estão prontos para executar cargas de trabalho.

Se, contudo, alguns dos nodes têm NotReady como o STATUS, isso pode significar que os nodes worker ainda não concluíram sua configuração. Aguarde cerca de cinco a dez minutos antes de voltar a executar kubectl get nodes e fazer a inspeção da nova saída. Se alguns nodes ainda têm NotReady como status, talvez seja necessário verificar e executar novamente os comandos nas etapas anteriores.

Agora que seu cluster foi verificado com sucesso, vamos fazer o scheduling de um exemplo de aplicativo Nginx no cluster.

Passo 7 — Executando Uma Aplicação no Cluster

Você pode fazer o deploy de qualquer aplicação containerizada no seu cluster. Para manter as coisas familiares, vamos fazer o deploy do Nginx utilizando Deployments e Services para ver como pode ser feito o deploy dessa aplicação no cluster. Você também pode usar os comandos abaixo para outros aplicativos em container, desde que você altere o nome da imagem do Docker e quaisquer flags relevantes (tais como ports e volumes).

Ainda no node master, execute o seguinte comando para criar um deployment chamado nginx:

kubectl run nginx --image=nginx --port 80

Um deployment é um tipo de objeto do Kubernetes que garante que há sempre um número especificado de pods em execução com base em um modelo definido, mesmo se o pod falhar durante o tempo de vida do cluster. O deployment acima irá criar um pod com um container do registro do Docker Nginx Docker Image.

A seguir, execute o seguinte comando para criar um serviço chamado nginx que irá expor o app publicamente. Ele fará isso por meio de um NodePort, um esquema que tornará o pod acessível através de uma porta arbitrária aberta em cada node do cluster:

kubectl expose deploy nginx --port 80 --target-port 80<^> --type NodePort

Services são outro tipo de objeto do Kubernetes que expõe serviços internos do cluster para os clientes, tanto internos quanto externos. Eles também são capazes de fazer balanceamento de solicitações para vários pods e são um componente integral no Kubernetes, interagindo frequentemente com outros componentes.

Execute o seguinte comando:

kubectl get services

Isso produzirá uma saída semelhante à seguinte:

OutputNAME         TYPE        CLUSTER-IP       EXTERNAL-IP           PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>                443/TCP        1d
nginx        NodePort    10.109.228.209   <none>                80:nginx_port/TCP   40m

A partir da terceira linha da saída acima, você pode obter a porta em que o Nginx está sendo executado. O Kubernetes atribuirá uma porta aleatória maior que 30000 automaticamente, enquanto garante que a porta já não esteja vinculada a outro serviço.

Para testar se tudo está funcionando, visite http://worker_1_ip:nginx_port ou http://worker_2_ip:nginx_port através de um navegador na sua máquina local. Você verá a familiar página de boas-vindas do Nginx.

Se você quiser remover o aplicativo Nginx, primeiro exclua o serviço nginx do node master:

kubectl delete service nginx

Execute o seguinte para garantir que o serviço tenha sido excluído:

kubectl get services

Você verá a seguinte saída:

OutputNAME         TYPE        CLUSTER-IP       EXTERNAL-IP           PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>                443/TCP        1d

Para excluir o deployment:

kubectl delete deployment nginx

Execute o seguinte para confirmar que isso funcionou:

kubectl get deployments

OutputNo resources found.

Conclusão

Neste guia, você configurou com sucesso um cluster do Kubernetes no CentOS 7 usando Kubeadm e Ansible para automação.

Se você está se perguntando o que fazer com o cluster, agora que ele está configurado, um bom próximo passo seria sentir-se confortável para implantar suas próprias aplicações e serviços no cluster. Aqui está uma lista de links com mais informações que podem orientá-lo no processo:

Dockerizing applications - lista exemplos que detalham como containerizar aplicações usando o Docker.
Pod Overview - descreve em detalhes como os Pods funcionam e seu relacionamento com outros objetos do Kubernetes. Os pods são onipresentes no Kubernetes, então compreendê-los facilitará seu trabalho.
Deployments Overview - fornece uma visão geral dos deployments. É útil entender como os controladores, como os deployments, funcionam, pois eles são usados com frequência em aplicações stateless para escalonamento e na recuperação automatizada de aplicações não íntegras.
Services Overview - cobre os serviços ou services, outro objeto frequentemente usado em clusters do Kubernetes. Entender os tipos de serviços e as opções que eles têm é essencial para executar aplicações stateless e stateful.

Outros conceitos importantes que você pode analisar são Volumes, Ingresses e Secrets, os quais são úteis ao realizar o deploy de aplicações em produção.

O Kubernetes tem muitas funcionalidades e recursos a oferecer. A Documentação Oficial do Kubernetes é o melhor lugar para aprender sobre conceitos, encontrar guias específicos de tarefas e procurar referências de API para vários objetos.

Por bsder

Como Criar um Cluster Kubernetes 1.11 Usando Kubeadm no Ubuntu 18.04

2018-09-11T00:44:21Z

O autor escolheu o Free and Open Source Fund para receber uma doação como parte do programa Write for DOnations.

Introdução

Neste guia, você vai configurar um cluster Kubernetes a partir do zero utilizando o Ansible e o Kubeadm, e a seguir fazer o deploy de uma aplicação Nginx containerizada nele.

Objetivos

Seu cluster irá incluir os seguintes recursos físicos:

Um node master

Dois nodes worker

Uma vez que o cluster esteja configurado, você fará o deploy do servidor web Nginx nele para assegurar que ele está executando as cargas de trabalho corretamente.

Pré-requisitos

Um par de chaves SSH em sua máquina Linux/macOS/BSD local. Se você não tiver usado chaves SSH antes, você pode aprender como configurá-las seguindo esta explicação em como configurar chaves SSH em sua máquina local.
Três servidores rodando Ubuntu 18.04 com pelo menos 1GB de RAM. Você deve ser capaz de fazer SSH em cada servidor como usuário root com o seu par de chaves SSH.
Ansible instalado em sua máquina local. Se você estiver executando Ubuntu 18.04 como seu SO, siga o a seção "Passo 1 - Instalando o Ansible" em Como instalar e configurar o Ansible no Ubuntu 18.04 para instalar o Ansible. Para instruções de instalações em outras plataformas como o macOS ou CentOS, siga a documentação oficial da instalação do Ansible.
Familiaridade com o playbooks do Ansible. Para uma revisão, verifique Configuration Management 101: Writing Ansible Playbooks.
Conhecimento de como lançar um container a partir de uma imagem Docker. Dê uma olhada no "Passo 5 — Executando um Container Docker" em Como instalar e usar o Docker no Ubuntu 18.04 se precisar relembrar.

Passo 1 — Configurando o Diretório da Área de Trabalho e o Arquivo de Inventário Ansible

Dos seus três servidores, um será o master com um IP exibido como master_ip. Os outros dois servidores serão workers e terão os IPs worker_1_ip e worker_2_ip.

Crie um diretório chamado ~/kube-cluster no diretório home de sua máquina local e faça um cd para dentro dele:

mkdir ~/kube-cluster
cd ~/kube-cluster

Crie um arquivo chamado ~/kube-cluster/hosts usando o nano ou o seu editor de textos favorito:

nano ~/kube-cluster/hosts

Adicione o seguinte texto ao arquivo, que irá especificar informações sobre a estrutura lógica do cluster:

~/kube-cluster/hosts


[masters]
master ansible_host=master_ip ansible_user=root

[workers]
worker1 ansible_host=worker_1_ip ansible_user=root
worker2 ansible_host=worker_2_ip ansible_user=root

[all:vars]
ansible_python_interpreter=/usr/bin/python3

No grupo masters, existe uma entrada de servidor chamada "master" que lista o IP do node master (master_ip) e especifica que o Ansible deve executar comandos remotos como root.

De maneira similar, no grupo workers, existem duas entradas para os servidores workers (worker_1_ip e worker_2_ip) que também especificam o ansible_user como root.

A última linha do arquivo diz ao Ansible para utilizar os intepretadores Python dos servidores remotos para suas operações de gerenciamento.

Salve e feche o arquivo depois de ter adicionado o texto.

Tendo configurado o inventário do servidor com grupos, vamos passar a instalar dependências no nível do sistema operacional e a criar definições de configuração.

Passo 2 — Criando um Usuário Não-Root em Todos os Servidores Remotos

Crie um arquivo chamado ~/kube-cluster/initial.yml na área de trabalho:

nano ~/kube-cluster/initial.yml

~/kube-cluster/initial.yml


- hosts: all
  become: yes
  tasks:
    - name: create the 'ubuntu' user
      user: name=ubuntu append=yes state=present createhome=yes shell=/bin/bash

    - name: allow 'ubuntu' to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        line: 'ubuntu ALL=(ALL) NOPASSWD: ALL'
        validate: 'visudo -cf %s'

    - name: set up authorized keys for the ubuntu user
      authorized_key: user=ubuntu key="{{item}}"
      with_file:
        - ~/.ssh/id_rsa.pub

Aqui está um detalhamento do que este playbook faz:

Cria um usuário não-root ubuntu.
Configura o arquivo sudoers para permitir o usuário ubuntu executar comandos sudo sem uma solicitação de senha.
Adiciona a chave pública em sua máquina local (normalmente ~/.ssh/id_rsa.pub) para a lista de chaves autorizadas do usuário remoto ubuntu. Isto o permitirá fazer SSH para dentro de cada servidor como usuário ubuntu.

Salve e feche o arquivo depois que tiver adicionado o texto.

Em seguida, rode o playbook localmente executando:

ansible-playbook -i hosts ~/kube-cluster/initial.yml

O comando será concluído dentro de dois a cinco minutos. Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [all] ****

TASK [Gathering Facts] ****
ok: [master]
ok: [worker1]
ok: [worker2]

TASK [create the 'ubuntu' user] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [allow 'ubuntu' user to have passwordless sudo] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [set up authorized keys for the ubuntu user] ****
changed: [worker1] => (item=ssh-rsa AAAAB3...)
changed: [worker2] => (item=ssh-rsa AAAAB3...)
changed: [master] => (item=ssh-rsa AAAAB3...)

PLAY RECAP ****
master                     : ok=5    changed=4    unreachable=0    failed=0   
worker1                    : ok=5    changed=4    unreachable=0    failed=0   
worker2                    : ok=5    changed=4    unreachable=0    failed=0

Agora que a configuração preliminar está completa, você pode passar para a instalação de dependências específicas do Kubernetes.

Step 3 — Instalando as Dependências do Kubernetes

Nesta seção, você irá instalar os pacotes no nível do sistema operacional necessários pelo Kubernetes com o gerenciador de pacotes do Ubuntu. Esses pacotes são:

Docker - um runtime de container. Este é o componente que executa seus containers. Suporte a outros runtimes como o rkt está em desenvolvimento ativo no Kubernetes.
kubeadm - uma ferramenta CLI que irá instalar e configurar os vários componentes de um cluster de uma maneira padrão.
kubelet - um serviço/programa de sistema que roda em todos os nodes e lida com operações no nível do node.
kubectl - uma ferramenta CLI usada para emitir comandos para o cluster através de seu servidor de API.

Crie um arquivo chamado ~/kube-cluster/kube-dependencies.yml na área de trabalho:

nano ~/kube-cluster/kube-dependencies.yml

Adicione os seguintes plays ao arquivo para instalar esses pacotes em seus servidores:

~/kube-cluster/kube-dependencies.yml


- hosts: all
  become: yes
  tasks:
   - name: install Docker
     apt:
       name: docker.io
       state: present
       update_cache: true

   - name: install APT Transport HTTPS
     apt:
       name: apt-transport-https
       state: present

   - name: add Kubernetes apt-key
     apt_key:
       url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
       state: present

   - name: add Kubernetes' APT repository
     apt_repository:
      repo: deb http://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: 'kubernetes'

   - name: install kubelet
     apt:
       name: kubelet
       state: present
       update_cache: true

   - name: install kubeadm
     apt:
       name: kubeadm
       state: present

- hosts: master
  become: yes
  tasks:
   - name: install kubectl
     apt:
       name: kubectl
       state: present

O primeiro play no playbook faz o seguinte:

Instala o Docker, o runtime de container.
Instala o apt-transport-https, permitindo que você adicione fontes HTTPS externas à sua lista de fontes do APT.
Adiciona a apt-key do repositório APT do Kubernetes para verificação de chave.
Adiciona o repositório APT do Kubernetes à lista de fontes do APT dos seus servidores remotos.
Instala kubelet e kubeadm.

O segundo play consiste de uma única tarefa que instala o kubectl no seu node master.

Salve e feche o arquivo quando você tiver terminado.

A seguir, rode o playbook executando localmente:

ansible-playbook -i hosts ~/kube-cluster/kube-dependencies.yml

Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [all] ****

TASK [Gathering Facts] ****
ok: [worker1]
ok: [worker2]
ok: [master]

TASK [install Docker] ****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [install APT Transport HTTPS] *****
ok: [master]
ok: [worker1]
changed: [worker2]

TASK [add Kubernetes apt-key] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [add Kubernetes' APT repository] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [install kubelet] *****
changed: [master]
changed: [worker1]
changed: [worker2]

TASK [install kubeadm] *****
changed: [master]
changed: [worker1]
changed: [worker2]

PLAY [master] *****

TASK [Gathering Facts] *****
ok: [master]

TASK [install kubectl] ******
ok: [master]

PLAY RECAP ****
master                     : ok=9    changed=5    unreachable=0    failed=0   
worker1                    : ok=7    changed=5    unreachable=0    failed=0  
worker2                    : ok=7    changed=5    unreachable=0    failed=0

Todas as dependências de sistema agora estão instaladas. Vamos configurar o node master e inicializar o cluster.

Passo 4 — Configurando o Node Master

Essa funcionalidade é fornecida pelos plugins de rede para pods. Para este cluster vamos utilizar o Flannel, uma opção estável e de bom desempenho.

Crie um playbook Ansible chamado master.yml em sua máquina local:

nano ~/kube-cluster/master.yml

Adicione o seguinte play ao arquivo para inicializar o cluster e instalar o Flannel:

~/kube-cluster/master.yml


- hosts: master
  become: yes
  tasks:
    - name: initialize the cluster
      shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt
      args:
        chdir: $HOME
        creates: cluster_initialized.txt

    - name: create .kube directory
      become: yes
      become_user: ubuntu
      file:
        path: $HOME/.kube
        state: directory
        mode: 0755

    - name: copy admin.conf to user's kube config
      copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/ubuntu/.kube/config
        remote_src: yes
        owner: ubuntu

    - name: install Pod network
      become: yes
      become_user: ubuntu
      shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml >> pod_network_setup.txt
      args:
        chdir: $HOME
        creates: pod_network_setup.txt

Aqui está um detalhamento deste play:

A primeira tarefa inicializa o cluster executando kubeadm init. A passagem do argumento --pod-network-cidr=10.244.0.0/16 especifica a sub-rede privada que os IPs do pod serão atribuídos. O Flannel utiliza a sub-rede acima por padrão; estamos dizendo ao kubeadm para utilizar a mesma sub-rede.
A segunda tarefa cria um diretório .kube em /home/ubuntu. Este diretório irá manter as informações de configuração tais como os arquivos de chaves do admin, que são requeridas para conectar no cluster, e o endereço da API do cluster.
A terceira tarefa copia o arquivo /etc/kubernetes/admin.conf que foi gerado a partir do kubeadm init para o diretório home do seu usuário não-root. Isso irá permitir que você utilize o kubectl para acessar o cluster recém-criado.
A última tarefa executa kubectl apply para instalar o Flannel. kubectl apply -f descriptor.[yml|json] é a sintaxe para dizer ao kubectl para criar os objetos descritos no arquivo descriptor.[yml|json]. O arquivo kube-flannel.yml contém as descrições dos objetos requeridos para a configuração do Flannel no cluster.

Salve e feche o arquivo quando você tiver terminado.

Rode o playbook localmente executando:

ansible-playbook -i hosts ~/kube-cluster/master.yml

Na conclusão, você verá uma saída semelhante à seguinte:

Output
PLAY [master] ****

TASK [Gathering Facts] ****
ok: [master]

TASK [initialize the cluster] ****
changed: [master]

TASK [create .kube directory] ****
changed: [master]

TASK [copy admin.conf to user's kube config] *****
changed: [master]

TASK [install Pod network] *****
changed: [master]

PLAY RECAP ****
master                     : ok=5    changed=4    unreachable=0    failed=0

Para verificar o status do node master, faça SSH nele com o seguinte comando:

ssh ubuntu@master_ip

Uma vez dentro do node master, execute:

kubectl get nodes

Agora você verá a seguinte saída:

OutputNAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    1d        v1.11.1

Passo 5 — Configurando os Nodes Worker

Navegue de volta para a sua área de trabalho e crie um playbook chamado workers.yml:

nano ~/kube-cluster/workers.yml

Adicione o seguinte texto ao arquivo para adicionar os workers ao cluster:

~/kube-cluster/workers.yml


- hosts: master
  become: yes
  gather_facts: false
  tasks:
    - name: get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

    - name: set join command
      set_fact:
        join_command: "{{ join_command_raw.stdout_lines[0] }}"


- hosts: workers
  become: yes
  tasks:
    - name: join cluster
      shell: "{{ hostvars['master'].join_command }} >> node_joined.txt"
      args:
        chdir: $HOME
        creates: node_joined.txt

Aqui está o que o playbook faz:

O primeiro play obtém o comando de junção que precisa ser executado nos nodes workers. Este comando estará no seguinte formato: kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>. Assim que obtiver o comando real com os valores apropriados de token e hash, a tarefa define isso como um fact para que o próximo play possa acessar essa informação.
O segundo play tem uma única tarefa que executa o comando de junção em todos os nodes worker. Na conclusão desta tarefa, os dois nodes worker farão parte do cluster.

Salve e feche o arquivo quando você tiver terminado.

Rode o playbook localmente executando:

ansible-playbook -i hosts ~/kube-cluster/workers.yml

Na conclusão, você verá uma saída semelhante à seguinte:

OutputPLAY [master] ****

TASK [get join command] ****
changed: [master]

TASK [set join command] *****
ok: [master]

PLAY [workers] *****

TASK [Gathering Facts] *****
ok: [worker1]
ok: [worker2]

TASK [join cluster] *****
changed: [worker1]
changed: [worker2]

PLAY RECAP *****
master                     : ok=2    changed=1    unreachable=0    failed=0   
worker1                    : ok=2    changed=1    unreachable=0    failed=0  
worker2                    : ok=2    changed=1    unreachable=0    failed=0

Step 6 — Verificando o Cluster

ssh ubuntu@master_ip

Em seguida, execute o seguinte comando para obter o status do cluster:

kubectl get nodes

Você verá uma saída semelhante à seguinte:

OutputNAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    1d        v1.11.1
worker1   Ready     <none>    1d        v1.11.1 
worker2   Ready     <none>    1d        v1.11.1

Se todos os seus nodes têm o valor Ready para o STATUS, significa que eles são parte do cluster e estão prontos para executar cargas de trabalho.

Agora que seu cluster foi verificado com sucesso, vamos fazer o scheduling de um exemplo de aplicativo Nginx no cluster.

Step 7 — Executando Uma Aplicação no Cluster

Ainda no node master, execute o seguinte comando para criar um deployment chamado nginx:

kubectl run nginx --image=nginx --port 80

kubectl expose deploy nginx --port 80 --target-port 80 --type NodePort

Services são outro tipo de objeto do Kubernetes que expõe serviços do cluster para os clientes, tanto internos quanto externos. Eles também são capazes de fazer balanceamento de solicitações para vários pods e são um componente integral no Kubernetes, interagindo frequentemente com outros componentes.

Execute o seguinte comando:

kubectl get services

Isso produzirá uma saída semelhante à seguinte:

OutputNAME         TYPE        CLUSTER-IP       EXTERNAL-IP           PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1        <none>                443/TCP             1d
nginx        NodePort    10.109.228.209   <none>                80:nginx_port/TCP   40m

Se você quiser remover o aplicativo Nginx, primeiro exclua o serviço nginx do node master:

kubectl delete service nginx

Execute o seguinte para garantir que o serviço tenha sido excluído:

kubectl get services

Você verá a seguinte saída:

[secondary label Output]
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP           PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>                443/TCP        1d

Para excluir o deployment:

kubectl delete deployment nginx

Execute o seguinte para confirmar que isso funcionou:

kubectl get deployments

OutputNo resources found.

Conclusão

Neste guia, você configurou com sucesso um cluster do Kubernetes no Ubuntu 18.04 usando Kubeadm e Ansible para automação.

Dockerizing applications - lista exemplos que detalham como containerizar aplicações usando o Docker.
Pod Overview - descreve em detalhes como os Pods funcionam e seu relacionamento com outros objetos do Kubernetes. Os pods são onipresentes no Kubernetes, então compreendê-los facilitará seu trabalho.
Deployments Overview - fornece uma visão geral dos deployments. É útil entender como os controladores, como os deployments, funcionam, pois eles são usados com frequência em aplicações stateless para escalonamento e na recuperação automatizada de aplicações não íntegras.
Services Overview - cobre os serviços ou services, outro objeto frequentemente usado em clusters do Kubernetes. Entender os tipos de serviços e as opções que eles têm é essencial para executar aplicações stateless e stateful.

Outros conceitos importantes que você pode analisar são Volumes, Ingresses e Secrets, os quais são úteis ao realizar o deploy de aplicações em produção.

Por bsder

How To Install Webmin on Debian 9

2018-09-07T21:44:48Z

Introduction

Webmin is a modern web control panel for any Linux machine that allows you to administer your server through a simple interface. With Webmin, you can change settings for common packages on the fly.

In this tutorial, you'll install and configure Webmin on your server and secure access to the interface with a valid certificate using Let's Encrypt. You'll then use Webmin to add new user accounts, and update all packages on your server from the dashboard.

Prerequisites

To complete this tutorial, you will need:

One Debian 9 server set up by following the Debian 9 initial server setup guide, including a sudo non-root user and a firewall.
Apache installed by following Step 1 of How To Install Linux, Apache, MariaDB, PHP (LAMP) stack on Debian 9. We'll use Apache to perform Let's Encrypt's domain verification.
A Fully-Qualified Domain Name (FQDN), with a DNS A record pointing to the IP address of your server. To configure this, follow these instructions on DNS hosting on DigitalOcean.

Step 1 — Installing Webmin

First, we need to add the Webmin repository so that we can easily install and update Webmin using our package manager. We do this by adding the repository to the /etc/apt/sources.list file.

Open the file in your editor:

sudo nano /etc/apt/sources.list

Then add this line to the bottom of the file to add the new repository:

/etc/apt/sources.list

 . . . 
deb http://download.webmin.com/download/repository sarge contrib

Save the file and exit the editor.

Next, add the Webmin PGP key so that your system will trust the new repository:

wget http://www.webmin.com/jcameron-key.asc
sudo apt-key add jcameron-key.asc

Next, update the list of packages to include the Webmin repository:

sudo apt update

Then install Webmin:

sudo apt install webmin

Once the installation finishes, you be presented with the following output:

OutputWebmin install complete. You can now login to 
https://your_server_ip:10000 as root with your 
root password, or as any user who can use `sudo`.

Please copy down this information, as you will need it for the next step.

Note: If you installed ufw during the prerequisite step, you will need to run the command sudo ufw allow 10000 in order to allow Webmin through the firewall. For extra security, you may want to configure your firewall to only allow access to this port from certain IP ranges.

Let's secure access to Webmin by adding a valid certificate.

Step 2 — Adding a Valid Certificate with Let's Encrypt

Webmin is already configured to use HTTPS, but it uses a self-signed, untrusted certificate. Let's replace it with a valid certificate from Let's Encrypt.

Navigate to https://your_domain:10000 in your web browser, replacing your_domain with the domain name you pointed at your server.

Note: When logging in for the first time, you will see an "Invalid SSL" error. This is because the server has generated a self-signed certificate. Allow the exception to continue so you can replace the self-signed certificate with one from Let's Encrypt.

You'll be presented with a login screen. Sign in with the non-root user you created while fulfilling the prerequisites for this tutorial.

Once you log in, the first screen you will see is the Webmin dashboard. Before you can apply a valid certificate, you have to set the server's hostname. Look for the System hostname field and click on the link to the right, as shown in the following figure:

This wil take you to the Hostname and DNS Client page. Locate the Hostname field, and enter your Fully-Qualified Domain Name into the field. Then press the Save button at the bottom of the page to apply the setting.

After you've set your hostname, click on Webmin on the left navigation bar, and then click on Webmin Configuration.

Then, select SSL Encryption from the list of icons, and then select the Let's Encrypt tab. You'll see a screen like the following figure:

Using this screen, you'll tell Webmin how to obtain and renew your certificate. Let's Encrypt certificates expire after 3 months, but we can instruct Webmin to automatically attempt to renew the Let's Encrypt certificate every month. Let's Encrypt looks for a verification file on our server, so we'll configure Webmin to place the verification file inside the folder /var/www/html, which is the folder that the Apache web server you configured in the prerequisites uses. Follow these steps to set up your certificate:

Fill in Hostnames for certificate with your FQDN.
For Website root directory for validation file, select the Other Directory button and enter /var/www/html.
For Months between automatic renewal section, deselect the Only renew manually option by typing 1 into the input box, and selecting the radio button to the left of the input box.
Click the Request Certificate button. After a few seconds, you will see a confirmation screen.

To use the new certificate, restart Webmin by clicking the back arrow in your browser, and clicking the Restart Webmin button. Wait around 30 seconds, and then reload the page and log in again. Your browser should now indicate that the certificate is valid.

Step 3 – Using Webmin

You've now set up a secured working instance of Webmin. Let's look at how to use it.

Webmin has many different modules that can control everything from the BIND DNS Server to something as simple as adding users to the system. Let's look at how to create a new user, and then explore how to update the operating system using Webmin.

Managing Users and Groups

Let's explore how to manage the users and groups on your server.

First, click the System tab, and then click the Users and Groups button. Then, from here, you can either add a user, manage a user, or add or manage a group.

Let's create a new user called deploy which can be used for hosting web applications. To add a user, click Create a new user, which is located at the top of the users table. This displays the Create User screen, where you can supply the username, password, groups and other options. Follow these instructions to create the user:

Fill in Username with deploy.
Select Automatic for User ID.
Fill in Real Name with a descriptive name like Deployment user.
For Home Directory, select Automatic.
For Shell, select /bin/bash from the dropdown list.
For Password, select Normal Password and type in a password of your choice.
For Primary Group, select New group with same name as user.
For Secondary Group, select sudo from the All groups list, and press the -> button to add the group to the in groups list.
Press Create to create this new user.

When creating a user, you can set options for password expiry, the user's shell, and whether or not they are allowed a home directory.

Next, let's look at how to install updates to our system.

Updating Packages

Webmin lets you update all of your packages through its user interface. To update all of your packages, first, go to the Dashboard link, and then locate the Package updates field. If there are updates available, you'll see a link that states the number of available updates, as shown in the following figure:

Click this link, and then press Update selected packages to start the update. You may be asked to reboot the server, which you can also do through the Webmin interface.

Conclusion

You now have a secured working instance of Webmin and you've used the interface to create a user and update packages. Webmin gives you access to many things you'd normally need to access through the console, and it organizes them in an intuitive way. For example, if you have Apache installed, you would find the configuration tab for it under Servers, and then Apache.

Explore the interface, or read the Official Webmin wiki to learn more about managing your system with Webmin.

How To Install Git on Debian 9

2018-09-07T22:40:33Z

Introduction

Software version control systems enable you to keep track of your software at the source level. With versioning tools, you can track changes, revert to previous stages, and branch to create alternate versions of files and directories.

Git is one of the most popular version control systems currently available. Many projects’ files are maintained in a Git repository, and sites like GitHub, GitLab, and Bitbucket help to facilitate software development project sharing and collaboration.

In this tutorial, we’ll install and configure Git on a Debian 9 server. We will cover how to install the software in two different ways, each of which have their own benefits depending on your specific needs.

Prerequisites

In order to complete this tutorial, you should have a non-root user with sudo privileges on an Debian 9 server. To learn how to achieve this setup, follow our Debian 9 initial server setup guide.

With your server and user set up, you are ready to begin.

Installing Git with Default Packages

Debian’s default repositories provide you with a fast method to install Git. Note that the version you install via these repositories may be older than the newest version currently available. If you need the latest release, consider moving to the next section of this tutorial to learn how to install and compile Git from source.

First, use the apt package management tools to update your local package index. With the update complete, you can download and install Git:

sudo apt update
sudo apt install git

You can confirm that you have installed Git correctly by running the following command:

git --version

Output
git version 2.11.0

With Git successfully installed, you can now move on to the Setting Up Git section of this tutorial to complete your setup.

Installing Git from Source

A more flexible method of installing Git is to compile the software from source. This takes longer and will not be maintained through your package manager, but it will allow you to download the latest release and will give you some control over the options you include if you wish to customize.

Before you begin, you need to install the software that Git depends on. This is all available in the default repositories, so we can update our local package index and then install the packages.

sudo apt update
sudo apt install make libssl-dev libghc-zlib-dev libcurl4-gnutls-dev libexpat1-dev gettext unzip

After you have installed the necessary dependencies, you can go ahead and get the version of Git you want by visiting the Git project’s mirror on GitHub, available via the following URL:

https://github.com/git/git

From here, be sure that you are on the master branch. Click on the Tags link and select your desired Git version. Unless you have a reason for downloading a release candidate (marked as rc) version, try to avoid these as they may be unstable.

Next, on the right side of the page, click on the Clone or download button, then right-click on Download ZIP and copy the link address that ends in .zip.

Back on your Debian 9 server, move into the tmp directory to download temporary files.

cd /tmp

From there, you can use the wget command to install the copied zip file link. We’ll specify a new name for the file: git.zip.

wget https://github.com/git/git/archive/v2.18.0.zip -O git.zip

Unzip the file that you downloaded and move into the resulting directory by typing:

unzip git.zip
cd git-*

Now, you can make the package and install it by typing these two commands:

make prefix=/usr/local all
sudo make prefix=/usr/local install

To ensure that the install was successful, you can type git --version and you should receive relevant output that specifies the current installed version of Git.

Now that you have Git installed, if you want to upgrade to a later version, you can clone the repository, and then build and install. To find the URL to use for the clone operation, navigate to the branch or tag that you want on the project's GitHub page and then copy the clone URL on the right side:

At the time of writing, the relevant URL is:

https://github.com/git/git.git

Change to your home directory, and use git clone on the URL you just copied:

cd ~
git clone https://github.com/git/git.git

This will create a new directory within your current directory where you can rebuild the package and reinstall the newer version, just like you did above. This will overwrite your older version with the new version:

cd git
make prefix=/usr/local all
sudo make prefix=/usr/local install

With this complete, you can be sure that your version of Git is up to date.

Setting Up Git

Now that you have Git installed, you should configure it so that the generated commit messages will contain your correct information.

This can be achieved by using the git config command. Specifically, we need to provide our name and email address because Git embeds this information into each commit we do. We can go ahead and add this information by typing:

git config --global user.name "Sammy"
git config --global user.email "sammy@domain.com"

We can see all of the configuration items that have been set by typing:

git config --list

Output
user.name=Sammy
user.email=sammy@domain.com
...

The information you enter is stored in your Git configuration file, which you can optionally edit by hand with a text editor like this:

nano ~/.gitconfig

~/.gitconfig contents

[user]
  name = Sammy
  email = sammy@domain.com

There are many other options that you can set, but these are the two essential ones needed. If you skip this step, you’ll likely see warnings when you commit to Git. This makes more work for you because you will then have to revise the commits you have done with the corrected information.

Conclusion

You should now have Git installed and ready to use on your system.

To learn more about how to use Git, check out these articles and series:

How To Set Up a Jupyter Notebook with Python 3 on Debian 9

2018-09-07T22:11:01Z

Introduction

Jupyter Notebook offers a command shell for interactive computing as a web application. The tool can be used with several languages, including Python, Julia, R, Haskell, and Ruby. It is often used for working with data, statistical modeling, and machine learning.

This tutorial will walk you through setting up Jupyter Notebook to run from a Debian 9 server, as well as teach you how to connect to and use the notebook. Jupyter notebooks (or simply notebooks) are documents produced by the Jupyter Notebook app which contain both computer code and rich text elements (paragraph, equations, figures, links, etc.) which aid in presenting and sharing reproducible research.

By the end of this guide, you will be able to run Python 3 code using Jupyter Notebook running on a remote server.

Prerequisites

In order to complete this guide, you should have a fresh Debian 9 server instance with a basic firewall and a non-root user with sudo privileges configured. You can learn how to set this up by running through our Initial Server Setup with Debian 9 guide.

Step 1 — Install Pip and Python Headers

To begin the process, we'll download and install all of the items we need from the Debian repositories. We will use the Python package manager pip to install additional components a bit later.

We first need to update the local apt package index and then download and install the packages:

sudo apt update

Next, install pip and the Python header files, which are used by some of Jupyter’s dependencies:

sudo apt install python3-pip python3-dev

Debian 9 (“Stretch”) comes preinstalled with Python 3.5.

We can now move on to setting up a Python virtual environment into which we’ll install Jupyter.

Step 2 — Create a Python Virtual Environment for Jupyter

Now that we have Python 3, its header files, and pip ready to go, we can create a Python virtual environment for easier management. We will install Jupyter into this virtual environment.

To do this, we first need access to the virtualenv command. We can install this with pip.

Upgrade pip and install the package by typing:

sudo -H pip3 install --upgrade pip
sudo -H pip3 install virtualenv

With virtualenv installed, we can start forming our environment. Create and move into a directory where we can keep our project files:

mkdir ~/myprojectdir
cd ~/myprojectdir

Within the project directory, create a Python virtual environment by typing:

virtualenv myprojectenv

This will create a directory called myprojectenv within your myprojectdir directory. Inside, it will install a local version of Python and a local version of pip. We can use this to install and configure an isolated Python environment for Jupyter.

Before we install Jupyter, we need to activate the virtual environment. You can do that by typing:

source myprojectenv/bin/activate

Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (myprojectenv)user@host:~/myprojectdir$.

You’re now ready to install Jupyter into this virtual environment.

Step 3 — Install Jupyter

With your virtual environment active, install Jupyter with the local instance of pip:

Note: When the virtual environment is activated (when your prompt has (myprojectenv) preceding it), use pip instead of pip3, even if you are using Python 3. The virtual environment's copy of the tool is always named pip, regardless of the Python version.

pip install jupyter

At this point, you’ve successfully installed all the software needed to run Jupyter. We can now start the notebook server.

Step 4 — Run Jupyter Notebook

You now have everything you need to run Jupyter Notebook! To run it, execute the following command:

jupyter notebook

A log of the activities of the Jupyter Notebook will be printed to the terminal. When you run Jupyter Notebook, it runs on a specific port number. The first notebook you run will usually use port 8888. To check the specific port number Jupyter Notebook is running on, refer to the output of the command used to start it:

Output[I 21:23:21.198 NotebookApp] Writing notebook server cookie secret to /run/user/1001/jupyter/notebook_cookie_secret
[I 21:23:21.361 NotebookApp] Serving notebooks from local directory: /home/sammy/myprojectdir
[I 21:23:21.361 NotebookApp] The Jupyter Notebook is running at:
[I 21:23:21.361 NotebookApp] http://localhost:8888/?token=1fefa6ab49a498a3f37c959404f7baf16b9a2eda3eaa6d72
[I 21:23:21.361 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 21:23:21.361 NotebookApp] No web browser found: could not locate runnable browser.
[C 21:23:21.361 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=1fefa6ab49a498a3f37c959404f7baf16b9a2eda3eaa6d72

If you are running Jupyter Notebook on a local Debian computer (not on a Droplet), you can simply navigate to the displayed URL to connect to Jupyter Notebook. If you are running Jupyter Notebook on a Droplet, you will need to connect to the server using SSH tunneling as outlined in the next section.

At this point, you can keep the SSH connection open and keep Jupyter Notebook running or can exit the app and re-run it once you set up SSH tunneling. Let's keep it simple and stop the Jupyter Notebook process. We will run it again once we have SSH tunneling working. To stop the Jupyter Notebook process, press CTRL+C, type Y, and hit ENTER to confirm. The following will be displayed:

Output[C 21:28:28.512 NotebookApp] Shutdown confirmed
[I 21:28:28.512 NotebookApp] Shutting down 0 kernels

We’ll now set up an SSH tunnel so that we can access the notebook.

Step 5 — Connect to the Server Using SSH Tunneling

In this section we will learn how to connect to the Jupyter Notebook web interface using SSH tunneling. Since Jupyter Notebook will run on a specific port on the server (such as :8888, :8889 etc.), SSH tunneling enables you to connect to the server’s port securely.

The next two subsections describe how to create an SSH tunnel from 1) a Mac or Linux and 2) Windows. Please refer to the subsection for your local computer.

SSH Tunneling with a Mac or Linux

If you are using a Mac or Linux, the steps for creating an SSH tunnel are similar to using SSH to log in to your remote server, except that there are additional parameters in the ssh command. This subsection will outline the additional parameters needed in the ssh command to tunnel successfully.

SSH tunneling can be done by running the following SSH command in a new local terminal window:

ssh -L 8888:localhost:8888 your_server_username@your_server_ip

The ssh command opens an SSH connection, but -L specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side (server). This means that whatever is running on the second port number (e.g. 8888) on the server will appear on the first port number (e.g. 8888) on your local computer.

Optionally change port 8888 to one of your choosing to avoid using a port already in use by another process.

server_username is your username (e.g. sammy) on the server which you created and your_server_ip is the IP address of your server.

For example, for the username sammy and the server address 203.0.113.0, the command would be:

ssh -L 8888:localhost:8888 sammy@203.0.113.0

If no error shows up after running the ssh -L command, you can move into your programming environment and run Jupyter Notebook:

jupyter notebook

You’ll receive output with a URL. From a web browser on your local machine, open the Jupyter Notebook web interface with the URL that starts with http://localhost:8888. Ensure that the token number is included, or enter the token number string when prompted at http://localhost:8888.

SSH Tunneling with Windows and Putty

If you are using Windows, you can create an SSH tunnel using Putty.

First, enter the server URL or IP address as the hostname as shown:

Next, click SSH on the bottom of the left pane to expand the menu, and then click Tunnels. Enter the local port number to use to access Jupyter on your local machine. Choose 8000 or greater to avoid ports used by other services, and set the destination as localhost:8888 where :8888 is the number of the port that Jupyter Notebook is running on.

Now click the Add button, and the ports should appear in the Forwarded ports list:

Finally, click the Open button to connect to the server via SSH and tunnel the desired ports. Navigate to http://localhost:8000 (or whatever port you chose) in a web browser to connect to Jupyter Notebook running on the server. Ensure that the token number is included, or enter the token number string when prompted at http://localhost:8000.

Step 6 — Using Jupyter Notebook

This section goes over the basics of using Jupyter Notebook. If you don’t currently have Jupyter Notebook running, start it with the jupyter notebook command.

You should now be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. This section will outline a few of the basic features to get you started using the notebook. Jupyter Notebook will show all of the files and folders in the directory it is run from, so when you’re working on a project make sure to start it from the project directory.

To create a new notebook file, select New > Python 3 from the top right pull-down menu:

This will open a notebook. We can now run Python code in the cell or change the cell to markdown. For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar. We can now write notes using Markdown and even include equations written in LaTeX by putting them between the $$ symbols. For example, type the following into the cell after changing it to markdown:

# Simple Equation

Let us now implement the following equation:
$$ y = x^2$$

where $x = 2$

To turn the markdown into rich text, press CTRL+ENTER, and the following should be the results:

You can use the markdown cells to make notes and document your code. Let's implement that simple equation and print the result. Click on the top cell, then press ALT+ENTER to add a cell below it. Enter the following code in the new cell.

x = 2
y = x**2
print(y)

To run the code, press CTRL+ENTER. You’ll receive the following results:

You now have the ability to import modules and use the notebook as you would with any other Python development environment!

Conclusion

Congratulations! You should now be able to write reproducible Python code and notes in Markdown using Jupyter Notebook. To get a quick tour of Jupyter Notebook from within the interface, select Help > User Interface Tour from the top navigation menu to learn more.

From here, you may be interested to read our series on Time Series Visualization and Forecasting.

How To Set Up an OpenVPN Server on Debian 9

2018-09-07T21:53:50Z

Introduction

Want to access the Internet safely and securely from your smartphone or laptop when connected to an untrusted network such as the WiFi of a hotel or coffee shop? A Virtual Private Network (VPN) allows you to traverse untrusted networks privately and securely as if you were on a private network. The traffic emerges from the VPN server and continues its journey to the destination.

When combined with HTTPS connections, this setup allows you to secure your wireless logins and transactions. You can circumvent geographical restrictions and censorship, and shield your location and any unencrypted HTTP traffic from the untrusted network.

OpenVPN is a full-featured, open-source Secure Socket Layer (SSL) VPN solution that accommodates a wide range of configurations. In this tutorial, you will set up an OpenVPN server on a Debian 9 server and then configure access to it from Windows, macOS, iOS and/or Android. This tutorial will keep the installation and configuration steps as simple as possible for each of these setups.

Note: If you plan to set up an OpenVPN server on a DigitalOcean Droplet, be aware that we, like many hosting providers, charge for bandwidth overages. For this reason, please be mindful of how much traffic your server is handling.

See this page for more info.

Prerequisites

To complete this tutorial, you will need access to a Debian 9 server to host your OpenVPN service. You will need to configure a non-root user with sudo privileges before you start this guide. You can follow our Debian 9 initial server setup guide to set up a user with appropriate permissions. The linked tutorial will also set up a firewall, which is assumed to be in place throughout this guide.

Additionally, you will need a separate machine to serve as your certificate authority (CA). While it’s technically possible to use your OpenVPN server or your local machine as your CA, this is not recommended as it opens up your VPN to some security vulnerabilities. Per the official OpenVPN documentation, you should place your CA on a standalone machine that’s dedicated to importing and signing certificate requests. For this reason, this guide assumes that your CA is on a separate Debian 9 server that also has a non-root user with sudo privileges and a basic firewall.

Please note that if you disable password authentication while configuring these servers, you may run into difficulties when transferring files between them later on in this guide. To resolve this issue, you could re-enable password authentication on each server. Alternatively, you could generate an SSH keypair for each server, then add the OpenVPN server’s public SSH key to the CA machine’s authorized_keys file and vice versa. See How to Set Up SSH Keys on Debian 9 for instructions on how to perform either of these solutions.

When you have these prerequisites in place, you can move on to Step 1 of this tutorial.

Step 1 — Installing OpenVPN and EasyRSA

To start off, update your VPN server’s package index and install OpenVPN. OpenVPN is available in Debian's default repositories, so you can use apt for the installation:

sudo apt update
sudo apt install openvpn

OpenVPN is a TLS/SSL VPN. This means that it utilizes certificates in order to encrypt traffic between the server and clients. To issue trusted certificates, you will set up your own simple certificate authority (CA). To do this, we will download the latest version of EasyRSA, which we’ll use to build our CA public key infrastructure (PKI), from the project’s official GitHub repository.

As mentioned in the prerequisites, we will build the CA on a standalone server. The reason for this approach is that, if an attacker were able to infiltrate your server, they would be able to access your CA private key and use it to sign new certificates, giving them access to your VPN. Accordingly, managing the CA from a standalone machine helps to prevent unauthorized users from accessing your VPN. Note, as well, that it’s recommended that you keep the CA server turned off when not being used to sign keys as a further precautionary measure.

To begin building the CA and PKI infrastructure, install the latest version of EasyRSA from the official GitHub project on both your CA machine and your OpenVPN server with the following command:

wget -P ~/ https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.4/EasyRSA-3.0.4.tgz

Then extract the tarball:

cd ~
tar xvf EasyRSA-3.0.4.tgz

You have successfully installed all the required software on your server and CA machine. Continue on to configure the variables used by EasyRSA and to set up a CA directory, from which you will generate the keys and certificates needed for your server and clients to access the VPN.

Step 2 — Configuring the EasyRSA Variables and Building the CA

EasyRSA comes installed with a configuration file which you can edit to define a number of variables for your CA.

On your CA machine, navigate to the EasyRSA directory:

cd ~/EasyRSA-3.0.4/

Inside this directory is a file named vars.example. Make a copy of this file, and name the copy vars without a file extension:

cp vars.example vars

Open this new file using your preferred text editor:

nano vars

Find the settings that set field defaults for new certificates. It will look something like this:

~/EasyRSA-3.0.4/vars

. . .

#set_var EASYRSA_REQ_COUNTRY    "US"
#set_var EASYRSA_REQ_PROVINCE   "California"
#set_var EASYRSA_REQ_CITY       "San Francisco"
#set_var EASYRSA_REQ_ORG        "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL      "me@example.net"
#set_var EASYRSA_REQ_OU         "My Organizational Unit"

. . .

Uncomment these lines and update the highlighted values to whatever you'd prefer, but do not leave them blank:

~/EasyRSA-3.0.4/vars

. . .

set_var EASYRSA_REQ_COUNTRY    "US"
set_var EASYRSA_REQ_PROVINCE   "NewYork"
set_var EASYRSA_REQ_CITY       "New York City"
set_var EASYRSA_REQ_ORG        "DigitalOcean"
set_var EASYRSA_REQ_EMAIL      "admin@example.com"
set_var EASYRSA_REQ_OU         "Community"

. . .

When you are finished, save and close the file.

Within the EasyRSA directory is a script called easyrsa which is called to perform a variety of tasks involved with building and managing the CA. Run this script with the init-pki option to initiate the public key infrastructure on the CA server:

./easyrsa init-pki

Output. . .
init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /home/sammy/EasyRSA-3.0.4/pki

After this, call the easyrsa script again, following it with the build-ca option. This will build the CA and create two important files — ca.crt and ca.key — which make up the public and private sides of an SSL certificate.

ca.crt is the CA’s public certificate file which, in the context of OpenVPN, the server and the client use to inform one another that they are part of the same web of trust and not someone performing a man-in-the-middle attack. For this reason, your server and all of your clients will need a copy of the ca.crt file.
ca.key is the private key which the CA machine uses to sign keys and certificates for servers and clients. If an attacker gains access to your CA and, in turn, your ca.key file, they will be able to sign certificate requests and gain access to your VPN, impeding its security. This is why your ca.key file should only be on your CA machine and that, ideally, your CA machine should be kept offline when not signing certificate requests as an extra security measure.

If you don’t want to be prompted for a password every time you interact with your CA, you can run the build-ca command with the nopass option, like this:

./easyrsa build-ca nopass

In the output, you’ll be asked to confirm the common name for your CA:

Output. . .
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:

The common name is the name used to refer to this machine in the context of the certificate authority. You can enter any string of characters for the CA’s common name but, for simplicity’s sake, press ENTER to accept the default name.

With that, your CA is in place and it’s ready to start signing certificate requests.

Step 3 — Creating the Server Certificate, Key, and Encryption Files

Now that you have a CA ready to go, you can generate a private key and certificate request from your server and then transfer the request over to your CA to be signed, creating the required certificate. You’re also free to create some additional files used during the encryption process.

Start by navigating to the EasyRSA directory on your OpenVPN server:

cd EasyRSA-3.0.4/

From there, run the easyrsa script with the init-pki option. Although you already ran this command on the CA machine, it’s necessary to run it here because your server and CA will have separate PKI directories:

./easyrsa init-pki

Then call the easyrsa script again, this time with the gen-req option followed by a common name for the machine. Again, this could be anything you like but it can be helpful to make it something descriptive. Throughout this tutorial, the OpenVPN server’s common name will simply be “server”. Be sure to include the nopass option as well. Failing to do so will password-protect the request file which could lead to permissions issues later on:

Note: If you choose a name other than “server” here, you will have to adjust some of the instructions below. For instance, when copying the generated files to the /etc/openvpn directory, you will have to substitute the correct names. You will also have to modify the /etc/openvpn/server.conf file later to point to the correct .crt and .key files.

./easyrsa gen-req server nopass

This will create a private key for the server and a certificate request file called server.req. Copy the server key to the /etc/openvpn/ directory:

sudo cp ~/EasyRSA-3.0.4/pki/private/server.key /etc/openvpn/

Using a secure method (like SCP, in our example below), transfer the server.req file to your CA machine:

scp ~/EasyRSA-3.0.4/pki/reqs/server.req sammy@your_CA_ip:/tmp

Next, on your CA machine, navigate to the EasyRSA directory:

cd EasyRSA-3.0.4/

Using the easyrsa script again, import the server.req file, following the file path with its common name:

./easyrsa import-req /tmp/server.req server

Then sign the request by running the easyrsa script with the sign-req option, followed by the request type and the common name. The request type can either be client or server, so for the OpenVPN server’s certificate request, be sure to use the server request type:

./easyrsa sign-req server server

In the output, you’ll be asked to verify that the request comes from a trusted source. Type yes then press ENTER to confirm this:

You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.

Request subject, to be signed as a server certificate for 3650 days:

subject=
    commonName                = server


Type the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes

If you encrypted your CA key, you’ll be prompted for your password at this point.

Next, transfer the signed certificate back to your VPN server using a secure method:

scp pki/issued/server.crt sammy@your_server_ip:/tmp

Before logging out of your CA machine, transfer the ca.crt file to your server as well:

scp pki/ca.crt sammy@your_server_ip:/tmp

Next, log back into your OpenVPN server and copy the server.crt and ca.crt files into your /etc/openvpn/ directory:

sudo cp /tmp/{server.crt,ca.crt} /etc/openvpn/

Then navigate to your EasyRSA directory:

cd EasyRSA-3.0.4/

From there, create a strong Diffie-Hellman key to use during key exchange by typing:

./easyrsa gen-dh

This may take a few minutes to complete. Once it does, generate an HMAC signature to strengthen the server's TLS integrity verification capabilities:

sudo openvpn --genkey --secret ta.key

When the command finishes, copy the two new files to your /etc/openvpn/ directory:

sudo cp ~/EasyRSA-3.0.4/ta.key /etc/openvpn/
sudo cp ~/EasyRSA-3.0.4/pki/dh.pem /etc/openvpn/

With that, all the certificate and key files needed by your server have been generated. You’re ready to create the corresponding certificates and keys which your client machine will use to access your OpenVPN server.

Step 4 — Generating a Client Certificate and Key Pair

Although you can generate a private key and certificate request on your client machine and then send it to the CA to be signed, this guide outlines a process for generating the certificate request on the server. The benefit of this is that we can create a script which will automatically generate client configuration files that contain all of the required keys and certificates. This lets you avoid having to transfer keys, certificates, and configuration files to clients and streamlines the process of joining the VPN.

We will generate a single client key and certificate pair for this guide. If you have more than one client, you can repeat this process for each one. Please note, though, that you will need to pass a unique name value to the script for every client. Throughout this tutorial, the first certificate/key pair is referred to as client1.

Get started by creating a directory structure within your home directory to store the client certificate and key files:

mkdir -p ~/client-configs/keys

Since you will store your clients’ certificate/key pairs and configuration files in this directory, you should lock down its permissions now as a security measure:

chmod -R 700 ~/client-configs

Next, navigate back to the EasyRSA directory and run the easyrsa script with the gen-req and nopass options, along with the common name for the client:

cd ~/EasyRSA-3.0.4/
./easyrsa gen-req client1 nopass

Press ENTER to confirm the common name. Then, copy the client1.key file to the /client-configs/keys/ directory you created earlier:

cp pki/private/client1.key ~/client-configs/keys/

Next, transfer the client1.req file to your CA machine using a secure method:

scp pki/reqs/client1.req sammy@your_CA_ip:/tmp

ssh sammy@your_CA_IP
cd EasyRSA-3.0.4/
./easyrsa import-req /tmp/client1.req client1

Then sign the request as you did for the server in the previous step. This time, though, be sure to specify the client request type:

./easyrsa sign-req client client1

At the prompt, enter yes to confirm that you intend to sign the certificate request and that it came from a trusted source:

OutputType the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes

Again, if you encrypted your CA key, you’ll be prompted for your password here.

This will create a client certificate file named client1.crt. Transfer this file back to the server:

scp pki/issued/client1.crt sammy@your_server_ip:/tmp

SSH back into your OpenVPN server and copy the client certificate to the /client-configs/keys/ directory:

cp /tmp/client1.crt ~/client-configs/keys/

Next, copy the ca.crt and ta.key files to the /client-configs/keys/ directory as well:

sudo cp EasyRSA-3.0.4/ta.key ~/client-configs/keys/
sudo cp /etc/openvpn/ca.crt ~/client-configs/keys/

With that, your server and client’s certificates and keys have all been generated and are stored in the appropriate directories on your server. There are still a few actions that need to be performed with these files, but those will come in a later step. For now, you can move on to configuring OpenVPN on your server.

Step 5 — Configuring the OpenVPN Service

Now that both your client and server’s certificates and keys have been generated, you can begin configuring the OpenVPN service to use these credentials.

Start by copying a sample OpenVPN configuration file into the configuration directory and then extract it in order to use it as a basis for your setup:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
sudo gzip -d /etc/openvpn/server.conf.gz

Open the server configuration file in your preferred text editor:

sudo nano /etc/openvpn/server.conf

Find the HMAC section by looking for the tls-auth directive. This line should already be uncommented, but if isn’t then remove the ";" to uncomment it. Below this line, add the key-direction parameter, set to "0":

/etc/openvpn/server.conf

tls-auth ta.key 0 # This file is secret
key-direction 0

Next, find the section on cryptographic ciphers by looking for the commented out cipher lines. The AES-256-CBC cipher offers a good level of encryption and is well supported. Again, this line should already be uncommented, but if it isn’t then just remove the ";" preceding it:

/etc/openvpn/server.conf

cipher AES-256-CBC

Below this, add an auth directive to select the HMAC message digest algorithm. For this, SHA256 is a good choice:

/etc/openvpn/server.conf

auth SHA256

Next, find the line containing a dh directive which defines the Diffie-Hellman parameters. Because of some recent changes made to EasyRSA, the filename for the Diffie-Hellman key may be different than what is listed in the example server configuration file. If necessary, change the file name listed here by removing the 2048 so it aligns with the key you generated in the previous step:

/etc/openvpn/server.conf

dh dh.pem

Finally, find the user and group settings and remove the ";" at the beginning of each to uncomment these lines:

/etc/openvpn/server.conf

user nobody
group nogroup

The changes you’ve made to the sample server.conf file up to this point are necessary in order for OpenVPN to function. The changes outlined below are optional, though they too are needed for many common use cases.

(Optional) Push DNS Changes to Redirect All Traffic Through the VPN

The settings above will create the VPN connection between the two machines, but will not force any connections to use the tunnel. If you wish to use the VPN to route all of your traffic, you will likely want to push the DNS settings to the client computers.

There are a few directives in the server.conf file which you must change in order to enable this functionality. Find the redirect-gateway section and remove the semicolon ";" from the beginning of the redirect-gateway line to uncomment it:

/etc/openvpn/server.conf

push "redirect-gateway def1 bypass-dhcp"

Just below this, find the dhcp-option section. Again, remove the ";" from in front of both of the lines to uncomment them:

/etc/openvpn/server.conf

push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

This will assist clients in reconfiguring their DNS settings to use the VPN tunnel for as the default gateway.

(Optional) Adjust the Port and Protocol

By default, the OpenVPN server uses port 1194 and the UDP protocol to accept client connections. If you need to use a different port because of restrictive network environments that your clients might be in, you can change the port option. If you are not hosting web content on your OpenVPN server, port 443 is a popular choice since it is usually allowed through firewall rules.

/etc/openvpn/server.conf

# Optional!
port 443

Oftentimes, the protocol is restricted to that port as well. If so, change proto from UDP to TCP:

/etc/openvpn/server.conf

# Optional!
proto tcp

If you do switch the protocol to TCP, you will need to change the explicit-exit-notify directive’s value from 1 to 0, as this directive is only used by UDP. Failing to do so while using TCP will cause errors when you start the OpenVPN service:

/etc/openvpn/server.conf

# Optional!
explicit-exit-notify 0

If you have no need to use a different port and protocol, it is best to leave these two settings as their defaults.

(Optional) Point to Non-Default Credentials

If you selected a different name during the ./build-key-server command earlier, modify the cert and key lines that you see to point to the appropriate .crt and .key files. If you used the default name, “server”, this is already set correctly:

/etc/openvpn/server.conf

cert server.crt
key server.key

When you are finished, save and close the file.

After going through and making whatever changes to your server’s OpenVPN configuration are required for your specific use case, you can begin making some changes to your server’s networking.

Step 6 — Adjusting the Server Networking Configuration

There are some aspects of the server’s networking configuration that need to be tweaked so that OpenVPN can correctly route traffic through the VPN. The first of these is IP forwarding, a method for determining where IP traffic should be routed. This is essential to the VPN functionality that your server will provide.

Adjust your server’s default IP forwarding setting by modifying the /etc/sysctl.conf file:

sudo nano /etc/sysctl.conf

Inside, look for the commented line that sets net.ipv4.ip_forward. Remove the "#" character from the beginning of the line to uncomment this setting:

/etc/sysctl.conf

net.ipv4.ip_forward=1

Save and close the file when you are finished.

To read the file and adjust the values for the current session, type:

sudo sysctl -p

Outputnet.ipv4.ip_forward = 1

If you followed the Debian 9 initial server setup guide listed in the prerequisites, you should have a UFW firewall in place. Regardless of whether you use the firewall to block unwanted traffic (which you almost always should do), for this guide you need a firewall to manipulate some of the traffic coming into the server. Some of the firewall rules must be modified to enable masquerading, an iptables concept that provides on-the-fly dynamic network address translation (NAT) to correctly route client connections.

Before opening the firewall configuration file to add the masquerading rules, you must first find the public network interface of your machine. To do this, type:

ip route | grep default

Your public interface is the string found within this command’s output that follows the word "dev". For example, this result shows the interface named eth0, which is highlighted below:

Output
default via 203.0.113.1 dev eth0 onlink

When you have the interface associated with your default route, open the /etc/ufw/before.rules file to add the relevant configuration:

sudo nano /etc/ufw/before.rules

UFW rules are typically added using the ufw command. Rules listed in the before.rules file, though, are read and put into place before the conventional UFW rules are loaded. Towards the top of the file, add the highlighted lines below. This will set the default policy for the POSTROUTING chain in the nat table and masquerade any traffic coming from the VPN. Remember to replace eth0 in the -A POSTROUTING line below with the interface you found in the above command:

/etc/ufw/before.rules

#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

# START OPENVPN RULES
# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0] 
# Allow traffic from OpenVPN client to eth0 (change to the interface you discovered!)
-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
COMMIT
# END OPENVPN RULES

# Don't delete these required lines, otherwise there will be errors
*filter
. . .

Save and close the file when you are finished.

Next, you need to tell UFW to allow forwarded packets by default as well. To do this, open the /etc/default/ufw file:

sudo nano /etc/default/ufw

Inside, find the DEFAULT_FORWARD_POLICY directive and change the value from DROP to ACCEPT:

/etc/default/ufw

DEFAULT_FORWARD_POLICY="ACCEPT"

Save and close the file when you are finished.

Next, adjust the firewall itself to allow traffic to OpenVPN. If you did not change the port and protocol in the /etc/openvpn/server.conf file, you will need to open up UDP traffic to port 1194. If you modified the port and/or protocol, substitute the values you selected here.

In case you forgot to add the SSH port when following the prerequisite tutorial, add it here as well:

sudo ufw allow 1194/udp
sudo ufw allow OpenSSH

After adding those rules, disable and re-enable UFW to restart it and load the changes from all of the files you've modified:

sudo ufw disable
sudo ufw enable

Your server is now configured to correctly handle OpenVPN traffic.

Step 7 — Starting and Enabling the OpenVPN Service

You're finally ready to start the OpenVPN service on your server. This is done using the systemd utility systemctl.

Start the OpenVPN server by specifying your configuration file name as an instance variable after the systemd unit file name. The configuration file for your server is called /etc/openvpn/server.conf, so add @server to end of your unit file when calling it:

sudo systemctl start openvpn@server

Double-check that the service has started successfully by typing:

sudo systemctl status openvpn@server

If everything went well, your output will look something like this:

Output● openvpn@server.service - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2016-05-03 15:30:05 EDT; 47s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
  Process: 5852 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid (code=exited, sta
 Main PID: 5856 (openvpn)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/system-openvpn.slice/openvpn@server.service
           └─5856 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/server.conf --writepid /run/openvpn/server.pid

You can also check that the OpenVPN tun0 interface is available by typing:

ip addr show tun0

This will output a configured interface:

Output4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever

After starting the service, enable it so that it starts automatically at boot:

sudo systemctl enable openvpn@server

Your OpenVPN service is now up and running. Before you can start using it, though, you must first create a configuration file for the client machine. This tutorial already went over how to create certificate/key pairs for clients, and in the next step we will demonstrate how to create an infrastructure that will generate client configuration files easily.

Step 8 — Creating the Client Configuration Infrastructure

Creating configuration files for OpenVPN clients can be somewhat involved, as every client must have its own config and each must align with the settings outlined in the server’s configuration file. Rather than writing a single configuration file that can only be used on one client, this step outlines a process for building a client configuration infrastructure which you can use to generate config files on-the-fly. You will first create a “base” configuration file then build a script which will allow you to generate unique client config files, certificates, and keys as needed.

Get started by creating a new directory where you will store client configuration files within the client-configs directory you created earlier:

mkdir -p ~/client-configs/files

Next, copy an example client configuration file into the client-configs directory to use as your base configuration:

cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf ~/client-configs/base.conf

Open this new file in your text editor:

nano ~/client-configs/base.conf

Inside, locate the remote directive. This points the client to your OpenVPN server address — the public IP address of your OpenVPN server. If you decided to change the port that the OpenVPN server is listening on, you will also need to change 1194 to the port you selected:

~/client-configs/base.conf

. . .
# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
remote your_server_ip 1194
. . .

Be sure that the protocol matches the value you are using in the server configuration:

~/client-configs/base.conf

proto udp

Next, uncomment the user and group directives by removing the ";" at the beginning of each line:

~/client-configs/base.conf

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nogroup

Find the directives that set the ca, cert, and key. Comment out these directives since you will add the certs and keys within the file itself shortly:

~/client-configs/base.conf

# SSL/TLS parms.
# See the server config file for more
# description.  It's best to use
# a separate .crt/.key file pair
# for each client.  A single ca
# file can be used for all clients.
#ca ca.crt
#cert client.crt
#key client.key

Mirror the cipher and auth settings that you set in the /etc/openvpn/server.conf file:

~/client-configs/base.conf

cipher AES-256-CBC
auth SHA256

Next, add the key-direction directive somewhere in the file. You must set this to "1" for the VPN to function correctly on the client machine:

~/client-configs/base.conf

key-direction 1

Finally, add a few commented out lines. Although you can include these directives in every client configuration file, you only need to enable them for Linux clients that ship with an /etc/openvpn/update-resolv-conf file. This script uses the resolvconf utility to update DNS information for Linux clients.

~/client-configs/base.conf

# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf

If your client is running Linux and has an /etc/openvpn/update-resolv-conf file, uncomment these lines from the client’s configuration file after it has been generated.

Save and close the file when you are finished.

Next, create a simple script that will compile your base configuration with the relevant certificate, key, and encryption files and then place the generated configuration in the ~/client-configs/files directory. Open a new file called make_config.sh within the ~/client-configs directory:

nano ~/client-configs/make_config.sh

Inside, add the following content, making sure to change sammy to that of your server’s non-root user account:

~/client-configs/make_config.sh

#!/bin/bash

# First argument: Client identifier

KEY_DIR=/home/sammy/client-configs/keys
OUTPUT_DIR=/home/sammy/client-configs/files
BASE_CONFIG=/home/sammy/client-configs/base.conf

cat ${BASE_CONFIG} \
    <(echo -e '<ca>') \
    ${KEY_DIR}/ca.crt \
    <(echo -e '</ca>\n<cert>') \
    ${KEY_DIR}/${1}.crt \
    <(echo -e '</cert>\n<key>') \
    ${KEY_DIR}/${1}.key \
    <(echo -e '</key>\n<tls-auth>') \
    ${KEY_DIR}/ta.key \
    <(echo -e '</tls-auth>') \
    > ${OUTPUT_DIR}/${1}.ovpn

Save and close the file when you are finished.

Before moving on, be sure to mark this file as executable by typing:

chmod 700 ~/client-configs/make_config.sh

This script will make a copy of the base.conf file you made, collect all the certificate and key files you’ve created for your client, extract their contents, append them to the copy of the base configuration file, and export all of this content into a new client configuration file. This means that, rather than having to manage the client’s configuration, certificate, and key files separately, all the required information is stored in one place. The benefit of this is that if you ever need to add a client in the future, you can just run this script to quickly create the config file and ensure that all the important information is stored in a single, easy-to-access location.

Please note that any time you add a new client, you will need to generate new keys and certificates for it before you can run this script and generate its configuration file. You will get some practice using this script in the next step.

Step 9 — Generating Client Configurations

If you followed along with the guide, you created a client certificate and key named client1.crt and client1.key, respectively, in Step 4. You can generate a config file for these credentials by moving into your ~/client-configs directory and running the script you made at the end of the previous step:

cd ~/client-configs
sudo ./make_config.sh client1

This will create a file named client1.ovpn in your ~/client-configs/files directory:

ls ~/client-configs/files

Outputclient1.ovpn

You need to transfer this file to the device you plan to use as the client. For instance, this could be your local computer or a mobile device.

While the exact applications used to accomplish this transfer will depend on your device's operating system and your personal preferences, a dependable and secure method is to use SFTP (SSH file transfer protocol) or SCP (Secure Copy) on the backend. This will transport your client's VPN authentication files over an encrypted connection.

Here is an example SFTP command using the client1.ovpn example which you can run from your local computer (macOS or Linux). It places the .ovpn file in your home directory:

sftp sammy@your_server_ip:client-configs/files/client1.ovpn ~/

Here are several tools and tutorials for securely transferring files from the server to a local computer:

Step 10 — Installing the Client Configuration

This section covers how to install a client VPN profile on Windows, macOS, Linux, iOS, and Android. None of these client instructions are dependent on one another, so feel free to skip to whichever is applicable to your device.

The OpenVPN connection will have the same name as whatever you called the .ovpn file. In regards to this tutorial, this means that the connection is named client1.ovpn, aligning with the first client file you generated.

Windows

Installing

Download the OpenVPN client application for Windows from OpenVPN's Downloads page. Choose the appropriate installer version for your version of Windows.

Note

OpenVPN needs administrative privileges to install.

After installing OpenVPN, copy the .ovpn file to:

C:\Program Files\OpenVPN\config

When you launch OpenVPN, it will automatically see the profile and makes it available.

You must run OpenVPN as an administrator each time it's used, even by administrative accounts. To do this without having to right-click and select Run as administrator every time you use the VPN, you must preset this from an administrative account. This also means that standard users will need to enter the administrator's password to use OpenVPN. On the other hand, standard users can't properly connect to the server unless the OpenVPN application on the client has admin rights, so the elevated privileges are necessary.

To set the OpenVPN application to always run as an administrator, right-click on its shortcut icon and go to Properties. At the bottom of the Compatibility tab, click the button to Change settings for all users. In the new window, check Run this program as an administrator.

Connecting

Each time you launch the OpenVPN GUI, Windows will ask if you want to allow the program to make changes to your computer. Click Yes. Launching the OpenVPN client application only puts the applet in the system tray so that you can connect and disconnect the VPN as needed; it does not actually make the VPN connection.

Once OpenVPN is started, initiate a connection by going into the system tray applet and right-clicking on the OpenVPN applet icon. This opens the context menu. Select client1 at the top of the menu (that's your client1.ovpn profile) and choose Connect.

A status window will open showing the log output while the connection is established, and a message will show once the client is connected.

Disconnect from the VPN the same way: Go into the system tray applet, right-click the OpenVPN applet icon, select the client profile and click Disconnect.

macOS

Installing

Tunnelblick is a free, open source OpenVPN client for macOS. You can download the latest disk image from the Tunnelblick Downloads page. Double-click the downloaded .dmg file and follow the prompts to install.

Towards the end of the installation process, Tunnelblick will ask if you have any configuration files. For simplicity, answer No and let Tunnelblick finish. Open a Finder window and double-click client1.ovpn. Tunnelblick will install the client profile. Administrative privileges are required.

Connecting

Launch Tunnelblick by double-clicking Tunnelblick in the Applications folder. Once Tunnelblick has been launched, there will be a Tunnelblick icon in the menu bar at the top right of the screen for controlling connections. Click on the icon, and then the Connect menu item to initiate the VPN connection. Select the client1 connection.

Linux

Installing

If you are using Linux, there are a variety of tools that you can use depending on your distribution. Your desktop environment or window manager might also include connection utilities.

The most universal way of connecting, however, is to just use the OpenVPN software.

On Ubuntu or Debian, you can install it just as you did on the server by typing:

sudo apt update
sudo apt install openvpn

On CentOS you can enable the EPEL repositories and then install it by typing:

sudo yum install epel-release
sudo yum install openvpn

Configuring

Check to see if your distribution includes an /etc/openvpn/update-resolv-conf script:

ls /etc/openvpn

Outputupdate-resolv-conf

Next, edit the OpenVPN client configuration file you transfered:

nano client1.ovpn

If you were able to find an update-resolv-conf file, uncomment the three lines you added to adjust the DNS settings:

client1.ovpn

script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

If you are using CentOS, change the group directive from nogroup to nobody to match the distribution's available groups:

client1.ovpn

group nobody

Save and close the file.

Now, you can connect to the VPN by just pointing the openvpn command to the client configuration file:

sudo openvpn --config client1.ovpn

This should connect you to your VPN.

iOS

Installing

From the iTunes App Store, search for and install OpenVPN Connect, the official iOS OpenVPN client application. To transfer your iOS client configuration onto the device, connect it directly to a computer.

The process of completing the transfer with iTunes is outlined here. Open iTunes on the computer and click on iPhone > apps. Scroll down to the bottom to the File Sharing section and click the OpenVPN app. The blank window to the right, OpenVPN Documents, is for sharing files. Drag the .ovpn file to the OpenVPN Documents window.

Now launch the OpenVPN app on the iPhone. You will receive a notification that a new profile is ready to import. Tap the green plus sign to import it.

Connecting

OpenVPN is now ready to use with the new profile. Start the connection by sliding the Connect button to the On position. Disconnect by sliding the same button to Off.

Note

The VPN switch under Settings cannot be used to connect to the VPN. If you try, you will receive a notice to only connect using the OpenVPN app.

Android

Installing

Open the Google Play Store. Search for and install Android OpenVPN Connect, the official Android OpenVPN client application.

You can transfer the .ovpn profile by connecting the Android device to your computer by USB and copying the file over. Alternatively, if you have an SD card reader, you can remove the device's SD card, copy the profile onto it and then insert the card back into the Android device.

Start the OpenVPN app and tap the menu to import the profile.

Then navigate to the location of the saved profile (the screenshot uses /sdcard/Download/) and select the file. The app will make a note that the profile was imported.

Connecting

To connect, simply tap the Connect button. You'll be asked if you trust the OpenVPN application. Choose OK to initiate the connection. To disconnect from the VPN, go back to the OpenVPN app and choose Disconnect.

Step 11 — Testing Your VPN Connection (Optional)

Note: This method for testing your VPN connection will only work if you opted to route all your traffic through the VPN in Step 5.

Once everything is installed, a simple check confirms everything is working properly. Without having a VPN connection enabled, open a browser and go to DNSLeakTest.

The site will return the IP address assigned by your internet service provider and as you appear to the rest of the world. To check your DNS settings through the same website, click on Extended Test and it will tell you which DNS servers you are using.

Now connect the OpenVPN client to your server’s VPN and refresh the browser. A completely different IP address (that of your VPN server) should now appear, and this is how you appear to the world. Again, DNSLeakTest's Extended Test will check your DNS settings and confirm you are now using the DNS resolvers pushed by your VPN.

Step 12 — Revoking Client Certificates

Occasionally, you may need to revoke a client certificate to prevent further access to the OpenVPN server.

To do so, navigate to the EasyRSA directory on your CA machine:

cd EasyRSA-3.0.4/

Next, run the easyrsa script with the revoke option, followed by the client name you wish to revoke:

./easyrsa revoke client2

This will ask you to confirm the revocation by entering yes:

OutputPlease confirm you wish to revoke the certificate with the following subject:

subject=
    commonName                = client2


Type the word 'yes' to continue, or any other input to abort.
  Continue with revocation: yes

After confirming the action, the CA will fully revoke the client’s certificate. However, your OpenVPN server currently has no way to check whether any clients’ certificates have been revoked and the client will still have access to the VPN. To correct this, create a certificate revocation list (CRL) on your CA machine:

./easyrsa gen-crl

This will generate a file called crl.pem. Securely transfer this file to your OpenVPN server:

scp ~/EasyRSA-3.0.4/pki/crl.pem sammy@your_server_ip:/tmp

On your OpenVPN server, copy this file into your /etc/openvpn/ directory:

sudo cp /tmp/crl.pem /etc/openvpn

Next, open the OpenVPN server configuration file:

sudo nano /etc/openvpn/server.conf

At the bottom of the file, add the crl-verify option, which will instruct the OpenVPN server to check the certificate revocation list that we've created each time a connection attempt is made:

/etc/openvpn/server.conf

crl-verify crl.pem

Save and close the file.

Finally, restart OpenVPN to implement the certificate revocation:

sudo systemctl restart openvpn@server

The client should no longer be able to successfully connect to the server using the old credential.

To revoke additional clients, follow this process:

Revoke the certificate with the ./easyrsa revoke client_name command
Generate a new CRL
Transfer the new crl.pem file to your OpenVPN server and copy it to the /etc/openvpn directory to overwrite the old list.
Restart the OpenVPN service.

You can use this process to revoke any certificates that you've previously issued for your server.

Conclusion

You are now securely traversing the internet protecting your identity, location, and traffic from snoopers and censors. If at this point you no longer need to issue certificates, it's recommended that you turn off your CA machine or otherwise disconnect it from the internet until you need to add or revoke certificates. This will help to prevent attackers from gaining access to your VPN.

To configure more clients, you only need to follow steps 4 and 9-11 for each additional device. To revoke access to clients, just follow step 12.

How To Install and Configure Nextcloud on Debian 9

2018-09-07T21:00:36Z

Introduction

Nextcloud, a fork of ownCloud, is a file sharing server that permits you to store your personal content, like documents and pictures, in a centralized location, much like Dropbox. The difference with Nextcloud is that all of its features are open-source. It also returns the control and security of your sensitive data back to you, thus eliminating the use of a third-party cloud hosting service.

In this tutorial, we will install and configure a Nextcloud instance on a Debian 9 server.

Prerequisites

In order to complete the steps in this guide, you will need the following:

A sudo user and firewall configured on your server: You can create a user with sudo privileges and set up a basic firewall by following the Debian 9 initial server setup guide.
(Optional) A domain name pointed to your server: We will be securing connections to the Nextcloud installation with TLS/SSL. Nextcloud can set up and manage a free, trusted SSL certificate from Let's Encrypt if your server has a domain name. If not, Nextcloud can set up a self-signed SSL certificate that can encrypt connections, but won't be trusted by default in web browsers. If you are using DigitalOcean, you can follow our guide on how to set up a domain name for your server if you intend to use Let's Encrypt.

Once you have completed the above steps, continue on to learn how to set up Nextcloud on your server.

Step 1 – Installing Nextcloud

We will be installing Nextcloud using the snappy packaging system. This packaging system, installable on Debian 9 through the default repositories, allows organizations to ship software, along with all associated dependencies and configuration, in a self-contained unit with automatic updates. This means that instead of installing and configuring a web and database server and then configuring the Nextcloud app to run on it, we can install the snap package which handles the underlying systems automatically.

To install and manage snap packages, we first need to install the snapd package on the server. Update the local package index for apt and then install the software by typing:

sudo apt update
sudo apt install snapd

Next, either log out and log back in again, or source the /etc/profile.d/apps-bin-path.sh script to add /snap/bin to your session's PATH variable:

source /etc/profile.d/apps-bin-path.sh

Once snapd is installed, you can download the Nextcloud snap package and install it on the system by typing:

sudo snap install nextcloud

The Nextcloud package will be downloaded and installed on your server. You can confirm that the installation process was successful by listing the changes associated with the snap:

snap changes nextcloud

OutputID   Status  Spawn               Ready               Summary
1    Done    today at 20:18 UTC  today at 20:18 UTC  Install "nextcloud" snap

The status and summary indicate that the installation was completed without any problems.

Getting Additional Information About the Nextcloud Snap

If you'd like some more information about the Nextcloud snap, there are a few commands that can be helpful.

The snap info command can show you the description, the Nextcloud management commands available, as well as the installed version and the snap channel being tracked:

snap info nextcloud

Snaps can define interfaces they support, which consist of a slot and plug that, when hooked together, gives the snap access to certain capabilities or levels of access. For instance, snaps that need to act as a network client must have the network interface. To see what snap "interfaces" this snap defines, type:

snap interfaces nextcloud

OutputSlot           Plug
:network       nextcloud
:network-bind  nextcloud
-              nextcloud:removable-media

To learn about all of the specific services and apps that this snap provides, you can take a look at the snap definition file by typing:

less /snap/nextcloud/current/meta/snap.yaml

This will allow you to see the individual components included within the snap, if you need help with debugging.

Configuring an Administrative Account

There are a few different ways you can configure the Nextcloud snap. In this guide, rather than creating an administrative user through the web interface, we will create one on the command line in order to avoid a small window where the administrator registration page would be accessible to anyone visiting your server's IP address or domain name.

To configure Nextcloud with a new administrator account, use the nextcloud.manual-install command. You must pass in a username and a password as arguments:

sudo -i nextcloud.manual-install sammy password

The following message indicates that Nextcloud has been configured correctly:

OutputNextcloud is not installed - only a limited number of commands are available
Nextcloud was successfully installed

Now that Nextcloud is installed, we need to adjust the trusted domains so that Nextcloud will respond to requests using the server's domain name or IP address.

Adjusting the Trusted Domains

When installing from the command line, Nextcloud restricts the host names that the instance will respond to. By default, the service only responds to requests made to the "localhost" hostname. We will be accessing Nextcloud through the server's domain name or IP address, so we'll need to adjust this setting to accept these type of requests.

You can view the current settings by querying the value of the trusted_domains array:

sudo -i nextcloud.occ config:system:get trusted_domains

Output
localhost

Currently, only localhost is present as the first value in the array. We can add an entry for our server's domain name or IP address by typing:

sudo -i nextcloud.occ config:system:set trusted_domains 1 --value=example.com

Output
System config value trusted_domains => 1 set to string example.com

If we query the trusted domains again, we will see that we now have two entries:

sudo -i nextcloud.occ config:system:get trusted_domains

Output
localhost
example.com

If you need to add another way of accessing the Nextcloud instance, you can add additional domains or addresses by rerunning the config:system:set command with an incremented index number (the "1" in the first command) and adjusting the --value.

Securing the Nextcloud Web Interface with SSL

Before we begin using Nextcloud, we need to secure the web interface.

If you have a domain name associated with your Nextcloud server, the Nextcloud snap can help you obtain and configure a trusted SSL certificate from Let's Encrypt. If your Nextcloud server does not have a domain name, Nextcloud can configure a self-signed certificate which will encrypt your web traffic but won't be able to verify the identity of your server.

With that in mind, follow the section below that matches your scenario.

Option 1: Setting Up SSL with Let's Encrypt

If you have a domain name associated with your Nextcloud server, the best option for securing your web interface is to obtain a Let's Encrypt SSL certificate.

Start by opening the ports in the firewall that Let's Encrypt uses to validate domain ownership. This will make your Nextcloud login page publicly accessible, but since we already have an administrator account configured, no one will be able to hijack the installation:

sudo ufw allow "WWW Full"

Next, request a Let's Encrypt certificate by typing:

sudo -i nextcloud.enable-https lets-encrypt

You will first be asked whether your server meets the conditions necessary to request a certificate from the Let's Encrypt service:

OutputIn order for Let's Encrypt to verify that you actually own the
domain(s) for which you're requesting a certificate, there are a
number of requirements of which you need to be aware:

1. In order to register with the Let's Encrypt ACME server, you must
   agree to the currently-in-effect Subscriber Agreement located
   here:

       https://letsencrypt.org/repository/

   By continuing to use this tool you agree to these terms. Please
   cancel now if otherwise.

2. You must have the domain name(s) for which you want certificates
   pointing at the external IP address of this machine.

3. Both ports 80 and 443 on the external IP address of this machine
   must point to this machine (e.g. port forwarding might need to be
   setup on your router).

Have you met these requirements? (y/n)

Type y to continue.

Next, you will be asked to provide an email address to use for recovery operations:

Output
Please enter an email address (for urgent notices or key recovery): your_email@domain.com

Finally, enter the domain name associated with your Nextcloud server:

Output
Please enter your domain name(s) (space-separated): example.com

Your Let's Encrypt certificate will be requested and, provided everything went well, the internal Apache instance will be restarted to immediately implement SSL:

OutputAttempting to obtain certificates... done
Restarting apache... done

You can now skip ahead to sign into Nextcloud for the first time.

Option 2: Setting Up SSL with a Self-Signed Certificate

If your Nextcloud server does not have a domain name, you can still secure the web interface by generating a self-signed SSL certificate. This certificate will allow access to the web interface over an encrypted connection, but will be unable to verify the identity of your server, so your browser will likely display a warning.

To generate a self-signed certificate and configure Nextcloud to use it, type:

sudo nextcloud.enable-https self-signed

OutputGenerating key and self-signed certificate... done
Restarting apache... done

The above output indicates that Nextcloud generated and enabled a self-signed certificate.

Now that the interface is secure, open the web ports in the firewall to allow access to the web interface:

sudo ufw allow "WWW Full"

You are now ready to log into Nextcloud for the first time.

Logging in to the Nextcloud Web Interface

Now that Nextcloud is configured, visit your server's domain name or IP address in your web browser:

https://example.com

Note: If you set up a self-signed SSL certificate, your browser may display a warning that the connection is insecure because the server's certificate is not signed by a recognized certificate authority. This is expected for self-signed certificates, so feel free to click through the warning to proceed to the site.

Since you have already configure an administrator account from the command line, you will be taken to the Nextcloud login page. Enter the credentials you created for the administrative user:

Click the Log in button to log in to the Nextcloud web interface.

The first time you enter, a window will be displayed with links to various Nextcloud clients that can be used to interact with and manage your Nextcloud instance:

Click through to download any clients you are interested in, or exit out of the window by clicking the X in the upper-right corner. You will be taken to the main Nextcloud interface, where you can begin to upload and manage files:

Your installation is now complete and secured. Feel free to explore the interface to get more familiarity with the features and functionality of your new system.

Conclusion

Nextcloud can replicate the capabilities of popular third-party cloud storage services. Content can be shared between users or externally with public URLs. The advantage of Nextcloud is that the information is stored securely in a place that you control.

Explore the interface and for additional functionality, install plugins using Nextcloud's app store.

How To Create a Self-Signed SSL Certificate for Nginx on Debian 9

2018-09-07T19:59:36Z

A previous version of this tutorial was written by Justin Ellingwood

Introduction

TLS, or transport layer security, and its predecessor SSL, which stands for secure sockets layer, are web protocols used to wrap normal traffic in a protected, encrypted wrapper.

Using this technology, servers can send traffic safely between the server and clients without the possibility of the messages being intercepted by outside parties. The certificate system also assists users in verifying the identity of the sites that they are connecting with.

In this guide, we will show you how to set up a self-signed SSL certificate for use with an Nginx web server on a Debian 9 server.

Note: A self-signed certificate will encrypt communication between your server and any clients. However, because it is not signed by any of the trusted certificate authorities included with web browsers, users cannot use the certificate to validate the identity of your server automatically.

A self-signed certificate may be appropriate if you do not have a domain name associated with your server and for instances where the encrypted web interface is not user-facing. If you do have a domain name, in many cases it is better to use a CA-signed certificate. To learn how to set up a free trusted certificate with the Let's Encrypt project, consult How to Secure Nginx with Let’s Encrypt on Debian 9.

Prerequisites

Before you begin, you should have a non-root user configured with sudo privileges. You can learn how to set up such a user account by following our initial server setup for Debian 9.

You will also need to have the Nginx web server installed. If you would like to install an entire LEMP (Linux, Nginx, MySQL, PHP) stack on your server, you can follow our guide on setting up LEMP on Debian 9.

If you just want the Nginx web server, you can instead follow our guide on installing Nginx on Debian 9.

When you have completed the prerequisites, continue below.

Step 1 — Creating the SSL Certificate

TLS/SSL works by using a combination of a public certificate and a private key. The SSL key is kept secret on the server. It is used to encrypt content sent to clients. The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

We can create a self-signed key and certificate pair with OpenSSL in a single command:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

You will be asked a series of questions. Before we go over that, let's take a look at what is happening in the command we are issuing:

openssl: This is the basic command line tool for creating and managing OpenSSL certificates, keys, and other files.
req: This subcommand specifies that we want to use X.509 certificate signing request (CSR) management. The "X.509" is a public key infrastructure standard that SSL and TLS adheres to for its key and certificate management. We want to create a new X.509 cert, so we are using this subcommand.
-x509: This further modifies the previous subcommand by telling the utility that we want to make a self-signed certificate instead of generating a certificate signing request, as would normally happen.
-nodes: This tells OpenSSL to skip the option to secure our certificate with a passphrase. We need Nginx to be able to read the file, without user intervention, when the server starts up. A passphrase would prevent this from happening because we would have to enter it after every restart.
-days 365: This option sets the length of time that the certificate will be considered valid. We set it for one year here.
-newkey rsa:2048: This specifies that we want to generate a new certificate and a new key at the same time. We did not create the key that is required to sign the certificate in a previous step, so we need to create it along with the certificate. The rsa:2048 portion tells it to make an RSA key that is 2048 bits long.
-keyout: This line tells OpenSSL where to place the generated private key file that we are creating.
-out: This tells OpenSSL where to place the certificate that we are creating.

As we stated above, these options will create both a key file and a certificate. We will be asked a few questions about our server in order to embed the information correctly in the certificate.

Fill out the prompts appropriately. The most important line is the one that requests the Common Name (e.g. server FQDN or YOUR name). You need to enter the domain name associated with your server or, more likely, your server's public IP address.

The entirety of the prompts will look something like this:

Output
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Bouncy Castles, Inc.
Organizational Unit Name (eg, section) []:Ministry of Water Slides
Common Name (e.g. server FQDN or YOUR name) []:server_IP_address
Email Address []:admin@your_domain.com

Both of the files you created will be placed in the appropriate subdirectories of the /etc/ssl directory.

While we are using OpenSSL, we should also create a strong Diffie-Hellman group, which is used in negotiating Perfect Forward Secrecy with clients.

We can do this by typing:

sudo openssl dhparam -out /etc/nginx/dhparam.pem 4096

This will take a while, but when it's done you will have a strong DH group at /etc/nginx/dhparam.pem that we can use in our configuration.

Step 2 — Configuring Nginx to Use SSL

We have created our key and certificate files under the /etc/ssl directory. Now we just need to modify our Nginx configuration to take advantage of these.

We will make a few adjustments to our configuration.

We will create a configuration snippet containing our SSL key and certificate file locations.
We will create a configuration snippet containing strong SSL settings that can be used with any certificates in the future.
We will adjust our Nginx server blocks to handle SSL requests and use the two snippets above.

This method of configuring Nginx will allow us to keep clean server blocks and put common configuration segments into reusable modules.

Creating a Configuration Snippet Pointing to the SSL Key and Certificate

First, let's create a new Nginx configuration snippet in the /etc/nginx/snippets directory.

To properly distinguish the purpose of this file, let's call it self-signed.conf:

sudo nano /etc/nginx/snippets/self-signed.conf

Within this file, we need to set the ssl_certificate directive to our certificate file and the ssl_certificate_key to the associated key. In our case, this will look like this:

/etc/nginx/snippets/self-signed.conf

ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

When you've added those lines, save and close the file.

Creating a Configuration Snippet with Strong Encryption Settings

Next, we will create another snippet that will define some SSL settings. This will set Nginx up with a strong SSL cipher suite and enable some advanced features that will help keep our server secure.

The parameters we will set can be reused in future Nginx configurations, so we will give the file a generic name:

sudo nano /etc/nginx/snippets/ssl-params.conf

To set up Nginx SSL securely, we will be using the recommendations by Remy van Elst on the Cipherli.st site. This site is designed to provide easy-to-consume encryption settings for popular software.

The suggested settings on the site linked to above offer strong security. Sometimes, this comes at the cost of greater client compatibility. If you need to support older clients, there is an alternative list that can be accessed by clicking the link on the page labelled "Yes, give me a ciphersuite that works with legacy / old software." That list can be substituted for the items copied below.

The choice of which config you use will depend largely on what you need to support. They both will provide great security.

For our purposes, we can copy the provided settings in their entirety. We just need to make a few small modifications.

First, we will add our preferred DNS resolver for upstream requests. We will use Google's for this guide.

Second, we will comment out the line that sets the strict transport security header. Before uncommenting this line, you should take take a moment to read up on HTTP Strict Transport Security, or HSTS, specifically about the "preload" functionality. Preloading HSTS provides increased security, but can have far reaching consequences if accidentally enabled or enabled incorrectly.

Copy the following into your ssl-params.conf snippet file:

/etc/nginx/snippets/ssl-params.conf

ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/dhparam.pem;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_timeout  10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# Disable strict transport security for now. You can uncomment the following
# line if you understand the implications.
# add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

Because we are using a self-signed certificate, SSL stapling will not be used. Nginx will output a warning but continue to operate correctly.

Save and close the file when you are finished.

Adjusting the Nginx Configuration to Use SSL

Now that we have our snippets, we can adjust our Nginx configuration to enable SSL.

We will assume in this guide that you are using a custom server block configuration file in the /etc/nginx/sites-available directory. We will use /etc/nginx/sites-available/example.com for this example. Substitute your configuration filename as needed.

Before we go any further, let's back up our current configuration file:

sudo cp /etc/nginx/sites-available/example.com /etc/nginx/sites-available/example.com.bak

Now, open the configuration file to make adjustments:

sudo nano /etc/nginx/sites-available/example.com

Inside, your server block probably begins similar to this:

/etc/nginx/sites-available/example.com

server {
    listen 80;
    listen [::]:80;

    server_name example.com www.example.com;

    root /var/www/example.com/html;
    index index.html index.htm index.nginx-debian.html;

    . . .
}

Your file may be in a different order, and instead of the root and index directives you may have some location, proxy_pass, or other custom configuration statements. This is ok, as we only need to update the listen directives and include our SSL snippets. We will be modifying this existing server block to serve SSL traffic on port 443, then create a new server block to respond on port 80 and automatically redirect traffic to port 443.

Note: We will use a 302 redirect until we have verified that everything is working properly. Afterwards, we can change this to a permanent 301 redirect.

In your existing configuration file, update the two listen statements to use port 443 and SSL, then include the two snippet files we created in previous steps:

/etc/nginx/sites-available/example.com

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    server_name example.com www.example.com;

    root /var/www/example.com/html;
    index index.html index.htm index.nginx-debian.html;

    . . .
}

Next, paste a second server block into the configuration file, after the closing bracket (}) of the first block:

/etc/nginx/sites-available/example.com

. . .
server {
    listen 80;
    listen [::]:80;

    server_name example.com www.example.com;

    return 302 https://$server_name$request_uri;
}

This is a bare-bones configuration that listens on port 80 and performs the redirect to HTTPS. Save and close the file when you are finished editing it.

Step 3 — Adjusting the Firewall

If you have the ufw firewall enabled, as recommended by the prerequisite guides, you'll need to adjust the settings to allow for SSL traffic. Luckily, Nginx registers a few profiles with ufw upon installation.

We can see the available profiles by typing:

sudo ufw app list

You should see a list like this:

OutputAvailable applications:
. . .
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
. . .

You can see the current setting by typing:

sudo ufw status

It will probably look like this, meaning that only HTTP traffic is allowed to the web server:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx HTTP                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

To additionally let in HTTPS traffic, we can allow the "Nginx Full" profile and then delete the redundant "Nginx HTTP" profile allowance:

sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'

Your status should look like this now:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

Step 4 — Enabling the Changes in Nginx

Now that we've made our changes and adjusted our firewall, we can restart Nginx to implement our new changes.

First, we should check to make sure that there are no syntax errors in our files. We can do this by typing:

sudo nginx -t

If everything is successful, you will get a result that looks like this:

Outputnginx: [warn] "ssl_stapling" ignored, issuer certificate not found
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Notice the warning in the beginning. As noted earlier, this particular setting throws a warning since our self-signed certificate can't use SSL stapling. This is expected and our server can still encrypt connections correctly.

If your output matches the above, your configuration file has no syntax errors. We can safely restart Nginx to implement our changes:

sudo systemctl restart nginx

Step 5 — Testing Encryption

Now, we're ready to test our SSL server.

Open your web browser and type https:// followed by your server's domain name or IP into the address bar:

https://server_domain_or_IP

Because the certificate we created isn't signed by one of your browser's trusted certificate authorities, you will likely see a scary looking warning like the one below (the following appears when using Google Chrome) :

This is expected and normal. We are only interested in the encryption aspect of our certificate, not the third party validation of our host's authenticity. Click "ADVANCED" and then the link provided to proceed to your host anyways:

You should be taken to your site. If you look in the browser address bar, you will see a lock with an "x" over it. In this case, this just means that the certificate cannot be validated. It is still encrypting your connection.

If you configured Nginx with two server blocks, automatically redirecting HTTP content to HTTPS, you can also check whether the redirect functions correctly:

http://server_domain_or_IP

If this results in the same icon, this means that your redirect worked correctly.

Step 6 — Changing to a Permanent Redirect

If your redirect worked correctly and you are sure you want to allow only encrypted traffic, you should modify the Nginx configuration to make the redirect permanent.

Open your server block configuration file again:

sudo nano /etc/nginx/sites-available/example.com

Find the return 302 and change it to return 301:

/etc/nginx/sites-available/example.com

    return 301 https://$server_name$request_uri;

Save and close the file.

Check your configuration for syntax errors:

sudo nginx -t

When you're ready, restart Nginx to make the redirect permanent:

sudo systemctl restart nginx

Conclusion

You have configured your Nginx server to use strong encryption for client connections. This will allow you serve requests securely, and will prevent outside parties from reading your traffic.

How To Install and Configure Postfix as a Send-Only SMTP Server on Debian 9

2018-09-07T18:19:23Z

Introduction

Postfix is a mail transfer agent (MTA), an application used to send and receive email. In this tutorial, you will install and configure Postfix so that it can be used to send emails by local applications only — that is, those installed on the same server as Postfix.

Why would you want to do that?

If you're already using a third-party email provider for sending and receiving emails, you do not need to run your own mail server. However, if you manage a cloud server on which you have installed applications that need to send email notifications, running a local, send-only SMTP server is a good alternative to using a third-party email service provider or running a full-blown SMTP server.

In this tutorial, you'll install and configure Postfix as a send-only SMTP server on Debian 9.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server, set up with the Debian 9 initial server setup tutorial, and a sudo non-root user.
A valid domain name, like example.com, pointing to your server. You can set that up by following these guidelines on managing DNS hosting on DigitalOcean.

Note that your server's hostname should match your domain or subdomain. You can verify the server's hostname by typing hostname at the command prompt. The output should match the name you gave the server when it was being created.

Step 1 — Installing Postfix

In this step, you'll learn how to install Postfix. You will need two packages: mailutils, which includes programs necessary for Postfix to function, and postfix itself.

First, update the package database:

sudo apt update

Next, install mailtuils:

sudo apt install mailutils

Finally, install postfix:

sudo apt install postfix

Near the end of the installation process, you will be presented with a window that looks like the one in the image below. The default option is Internet Site. That's the recommended option for this tutorial, so press TAB, then ENTER.

After that, you'll get another window just like the one in the next image. The System mail name should be the same as the name you assigned to the server when you were creating it. If it shows a subdomain like subdomain.example.com, change it to just example.com. When you've finished, press TAB, then ENTER.

You now have Postfix installed and are ready to modify its configuration settings.

Step 2 — Configuring Postfix

In this step, you'll configure Postfix to process requests to send emails only from the server on which it is running, i.e. from localhost.

For that to happen, Postfix needs to be configured to listen only on the loopback interface, the virtual network interface that the server uses to communicate internally. To make the change, open the main Postfix configuration file using nano or your favorite text editor:

sudo nano /etc/postfix/main.cf

With the file open, scroll down until you see the following section:

/etc/postfix/main.cf

. . .
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
. . .

Change the line that reads inet_interfaces = all to inet_interfaces = loopback-only:

/etc/postfix/main.cf

. . .
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = loopback-only
. . .

Another directive you'll need to modify is mydestination, which is used to specify the list of domains that are delivered via the local_transport mail delivery transport. By default, the values are similar to these:

/etc/postfix/main.cf. . .
mydestination = $myhostname, example.com, localhost.com, , localhost
. . .

The recommended defaults for this directive are given in the code block below, so modify yours to match:

/etc/postfix/main.cf. . .
mydestination = $myhostname, localhost.$your_domain, $your_domain
. . .

Save and close the file.

Note: If you're hosting multiple domains on a single server, the other domains can also be passed to Postfix using the mydestination directive. However, to configure Postfix in a manner that scales and that does not present issues for such a setup involves additional configurations that are beyond the scope of this article.

Finally, restart Postfix.

sudo systemctl restart postfix

Step 3 — Testing the SMTP Server

In this step, you'll test whether Postfix can send emails to an external email account using the mail command, which is part of the mailutils package you installed in Step 1.

To send a test email, type:

echo "This is the body of the email" | mail -s "This is the subject line" your_email_address

In performing your own test(s), you may use the body and subject line text as-is, or change them to your liking. However, in place of your_email_address, use a valid email address. The domain part can be gmail.com, fastmail.com, yahoo.com, or any other email service provider that you use.

Now check the email address where you sent the test message. You should see the message in your Inbox. If not, check your Spam folder.

Note that with this configuration, the address in the From field for the test emails you send will be sammy@example.com, where sammy is your Linux username and the domain is the server's hostname. If you change your username, the From address will also change.

Step 4 — Forwarding System Mail

The last thing we want to set up is forwarding, so you'll get emails sent to root on the system at your personal, external email address.

To configure Postfix so that system-generated emails will be sent to your email address, you need to edit the /etc/aliases file:

sudo nano /etc/aliases

The full contents of the file on a default installation of Debian 9 are as follows:

/etc/aliases

mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
abuse: root
noc: root
security: root

The postmaster: root setting ensures that system-generated emails are sent to the root user. You want to edit these settings so these emails are rerouted to your email address. To accomplish that, edit the file so that it reads:

/etc/aliases

mailer-daemon: postmaster
postmaster:    root
root:          your_email_address
. . .

Replace your_email_address with your personal email address. When finished, save and close the file. For the change to take effect, run the following command:

sudo newaliases

You can test that it works by sending an email to the root account using:

echo "This is the body of the email" | mail -s "This is the subject line" root

You should receive the email at your email address. If not, check your Spam folder.

Conclusion

That's all it takes to set up a send-only email server using Postfix. You may want to take some additional steps to protect your domain from spammers, however.

If you want to receive notifications from your server at a single address, then having emails marked as Spam is less of an issue because you can create a whitelist workaround. However, if you want to send emails to potential site users (such as confirmation emails for a message board sign-up), you should definitely set up SPF records and DKIM so your server's emails are more likely to be seen as legitimate.

If configured correctly, these steps make it difficult to send Spam with an address that appears to originate from your domain. Taking these additional configuration steps will also make it more likely for common mail providers to see emails from your server as legitimate.

How to Install Hadoop in Stand-Alone Mode on Debian 9

2018-09-07T16:57:50Z

Introduction

Hadoop is a Java-based programming framework that supports the processing and storage of extremely large datasets on a cluster of inexpensive machines. It was the first major open source project in the big data playing field and is sponsored by the Apache Software Foundation.

Hadoop is comprised of four main layers:

Hadoop Common is the collection of utilities and libraries that support other Hadoop modules.
HDFS, which stands for Hadoop Distributed File System, is responsible for persisting data to disk.
YARN, short for Yet Another Resource Negotiator, is the "operating system" for HDFS.
MapReduce is the original processing model for Hadoop clusters. It distributes work within the cluster or map, then organizes and reduces the results from the nodes into a response to a query. Many other processing models are available for the 3.x version of Hadoop.

Hadoop clusters are relatively complex to set up, so the project includes a stand-alone mode which is suitable for learning about Hadoop, performing simple operations, and debugging.

In this tutorial, you'll install Hadoop in stand-alone mode and run one of the example example MapReduce programs it includes to verify the installation.

Before you begin, you might also like to take a look at An Introduction to Big Data Concepts and Terminology or An Introduction to Hadoop

Prerequisites

To follow this tutorial, you will need:

A Debian 9 server with a non-root user with sudo privileges and a firewall, which you can set up by following the Initial Server Setup with Debian 9 tutorial.
Java installed by following How to Install Java with Apt on Debian 9. You can use OpenJDK for this tutorial.
The JAVA_HOME environment variable set in /etc/environment, as shown in How to Install Java with Apt on Debian 9. Hadoop requires this variable to be set.

Step 1 — Installing Hadoop

To install Hadoop, first visit the Apache Hadoop Releases page to find the most recent stable release.

Navigate to binary for the release you’d like to install. In this guide, we’ll install Hadoop 3.0.3.

On the next page, right-click and copy the link to the release binary.

On your server, use wget to fetch it:

wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.0.3/hadoop-3.0.3.tar.gz

Note: The Apache website will direct you to the best mirror dynamically, so your URL may not match the URL above.

In order to ensure that the file you downloaded hasn't been altered, do a quick check using SHA-256. Return to the releases page, then right-click and copy the link to the checksum file for the release binary you downloaded:

Again, use wget on your server to download the file:

wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.0.3/hadoop-3.0.3.tar.gz.mds

Then run the verification:

sha256sum hadoop-3.0.3.tar.gz

Output
db96e2c0d0d5352d8984892dfac4e27c0e682d98a497b7e04ee97c3e2019277a  hadoop-3.0.3.tar.gz

Compare this value with the SHA-256 value in the .mds file:

cat hadoop-3.0.3.tar.gz.mds | grep SHA256

~/hadoop-3.0.3.tar.gz.mds

...
SHA256 = DB96E2C0 D0D5352D 8984892D FAC4E27C 0E682D98 A497B7E0 4EE97C3E 2019277A

You can safely ignore the difference in case and the spaces. The output of the command you ran against the file we downloaded from the mirror should match the value in the file you downloaded from apache.org.

Now that you've verified that the file wasn't corrupted or changed, use the tar command with the -x flag to extract, -z to uncompress, -v for verbose output, and -f to specify that you're extracting the archive from a file. Use tab-completion or substitute the correct version number in the command below:

tar -xzvf hadoop-3.0.3.tar.gz

Finally, move the extracted files into /usr/local, the appropriate place for locally installed software. Change the version number, if needed, to match the version you downloaded.

sudo mv hadoop-3.0.3 /usr/local/hadoop

With the software in place, we're ready to configure its environment.

Step 3 — Running Hadoop

Let's make sure Hadoop runs. Execute the following command to launch Hadoop and display its help options:

/usr/local/hadoop/bin/hadoop

You'll see the following output, which lets you know you've successfully configured Hadoop to run in stand-alone mode.

OutputUsage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
 or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
  where CLASSNAME is a user-provided Java class

  OPTIONS is none or any of:

--config dir                     Hadoop config directory
--debug                          turn on shell script debug mode
--help                           usage information
buildpaths                       attempt to add class files from build tree
hostnames list[,of,host,names]   hosts to use in slave mode
hosts filename                   list of hosts to use in slave mode
loglevel level                   set the log4j level for this command
workers                          turn on worker mode

  SUBCOMMAND is one of:
. . .

We'll ensure that it is functioning properly by running the example MapReduce program it ships with. To do so, create a directory called input in your home directory and copy Hadoop's configuration files into it to use those files as our data.

mkdir ~/input
cp /usr/local/hadoop/etc/hadoop/*.xml ~/input

Next, we'll run the MapReduce hadoop-mapreduce-examples program, a Java archive with several options. We'll invoke its grep program, one of the many examples included in hadoop-mapreduce-examples, followed by the input directory, input and the output directory grep_example. The MapReduce grep program will count the matches of a literal word or regular expression. Finally, we'll supply the regular expression allowed[.]* to find occurrences of the word allowed within or at the end of a declarative sentence. The expression is case-sensitive, so we wouldn't find the word if it were capitalized at the beginning of a sentence.

Execute the following command:

/usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar grep ~/input ~/grep_example 'allowed[.]*'

When the task completes, it provides a summary of what has been processed and errors it has encountered, but this doesn't contain the actual results:

Output . . .
        File System Counters
        FILE: Number of bytes read=1330690
        FILE: Number of bytes written=3128841
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=2
        Map output records=2
        Map output bytes=33
        Map output materialized bytes=43
        Input split bytes=115
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=43
        Reduce input records=2
        Reduce output records=2
        Spilled Records=4
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=3
        Total committed heap usage (bytes)=478150656
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=147
    File Output Format Counters
        Bytes Written=34

The results are stored in the ~/grep_example directory.

If this output directory already exists, the program will fail, and rather than seeing the summary, you'll see something like this:

Output . . .
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:158)

Check the results by running cat on the output directory:

cat ~/grep_example/*

You'll see this output:

Output19  allowed.
1   allowed

The MapReduce task found 19 occurrences of the word allowed followed by a period and one occurrence where it was not. Running the example program has verified that our stand-alone installation is working properly and that non-privileged users on the system can run Hadoop for exploration or debugging.

Conclusion

In this tutorial, we've installed Hadoop in stand-alone mode and verified it by running an example program it provided. To learn how to write your own MapReduce programs, visit Apache Hadoop's MapReduce tutorial which walks through the code behind the example you used in this tutorial. When you're ready to set up a cluster, see the Apache Foundation Hadoop Cluster Setup guide.

How To Rewrite URLs with mod_rewrite for Apache on Debian 9

2018-09-07T16:40:21Z

Introduction

Apache's mod_rewrite module lets you rewrite URLs in a cleaner fashion, translating human-readable paths into code-friendly query strings. It also lets you rewrite URLs based on conditions.

An .htaccess file lets you create and apply rewrite rules without accessing server configuration files. By placing the .htaccess file in the root of your web site, you can manage rewrites on a per-site or per-directory basis.

In this tutorial, you'll enable mod_rewrite and use .htaccess files to create a basic URL redirection, and then explore a couple of advanced use cases.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server set up by following the Debian 9 initial server setup guide, including a sudo non-root user and a firewall.
Apache installed by following Steps 1 and 2 of How To Install the Apache Web Server on Debian 9.

Step 1 — Enabling mod_rewrite

In order for Apache to understand rewrite rules, we first need to activate mod_rewrite. It's already installed, but it's disabled on a default Apache installation. Use the a2enmod command to enable the module:

sudo a2enmod rewrite

This will activate the module or alert you that the module is already enabled. To put these changes into effect, restart Apache:

sudo systemctl restart apache2

mod_rewrite is now fully enabled. In the next step we will set up an .htaccess file that we'll use to define rewrite rules for redirects.

Step 2 — Setting Up .htaccess

An .htaccess file allows us to modify our rewrite rules without accessing server configuration files. For this reason, .htaccess is critical to your web application's security. The period that precedes the filename ensures that the file is hidden.

Note: Any rules that you can put in an .htaccess file can also be put directly into server configuration files. In fact, the official Apache documentation recommends using server configuration files instead of .htaccess thanks to faster processing times.

However, in this simple example, the performance increase will be negligible. Additionally, setting rules in .htaccess is convenient, especially with multiple websites on the same server. It does not require a server restart for changes to take effect or root privileges to edit rules, simplifying maintenance and the process of making changes with an unprivileged account. Popular open-source software like Wordpress and Joomla relies on .htaccess files to make modifications and additional rules on demand.

Before you start using .htaccess files, you'll need to set up and secure a few more settings.

By default, Apache prohibits using an .htaccess file to apply rewrite rules, so first you need to allow changes to the file. Open the default Apache configuration file using nano or your favorite text editor:

sudo nano /etc/apache2/sites-available/000-default.conf

Inside that file, you will find a <VirtualHost *:80> block starting on the first line. Inside of that block, add the following new block so your configuration file looks like the following. Make sure that all blocks are properly indented.

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
    <Directory /var/www/html>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    . . .
</VirtualHost>

Save and close the file.

Check your configuration:

sudo apache2ctl configtest

If there are no errors, restart Apache to put your changes into effect:

sudo systemctl restart apache2

Now, create an .htaccess file in the web root:

sudo nano /var/www/html/.htaccess

Add this line at the top of the new file to activate the rewrite engine.

/var/www/html/.htaccess

RewriteEngine on

Save the file and exit.

You now have an operational .htaccess file that you can use to govern your web application's routing rules. In the next step, we will create sample website files that we'll use to demonstrate rewrite rules.

Step 3 — Configuring URL Rewrites

Here, we will set up a basic URL rewrite which converts pretty URLs into actual paths to pages. Specifically, we will allow users to access http://your_server_ip/about, and display a page called about.html.

Begin by creating a file named about.html in the web root:

sudo nano /var/www/html/about.html

Copy the following HTML code into the file, then save and close it.

/var/www/html/about.html

<html>
    <head>
        <title>About Us</title>
    </head>
    <body>
        <h1>About Us</h1>
    </body>
</html>

You can access this page at http://your_server_ip/about.html, but notice that if you try to access http://your_server_ip/about, you will see a 404 Not Found error. To access the page using /about instead, we'll create a rewrite rule.

All RewriteRules follow this format:

General RewriteRule structure

RewriteRule pattern substitution [flags]

RewriteRule specifies the directive.
pattern is a regular expression that matches the desired string from the URL, which is what the viewer types in the browser.
substitution is the path to the actual URL, i.e. the path of the file Apache serves.
flags are optional parameters that can modify how the rule works.

Let's create our URL rewrite rule. Open up the .htaccess file:

sudo nano /var/www/html/.htaccess

After the first line, add the following RewriteRule and save the file:

/var/www/html/.htaccess

RewriteEngine on
RewriteRule ^about$ about.html [NC]

In this case, ^about$ is the pattern, about.html is the substitution, and [NC] is a flag. Our example uses a few characters with special meaning:

^ indicates the start of the URL, after your_server_ip/.
$ indicates the end of the URL.
about matches the string "about".
about.html is the actual file that the user accesses.
[NC] is a flag that makes the rule case insensitive.

You can now access http://your_server_ip/about in your browser. In fact, with the rule shown above, the following URLs will also point to about.html:

http://your_server_ip/about, because of the rule definition.
http://your_server_ip/About, because the rule is case insensitive.
http://your_server_ip/about.html, because the original filename will always work.

However, the following will not work:

http://your_server_ip/about/, because the rule explicitly states that there may be nothing after about, since the $ character appears after about.
http://your_server_ip/contact, because it won't match the about string in the rule.

You now have an operational .htaccess file with a basic rule that you can modify and extend to your needs. In the following sections, we will show two additional examples of commonly used directives.

Example 1 — Simplifying Query Strings with RewriteRule

Web applications often make use of query strings, which are appended to a URL using a question mark (?) after the address. Separate parameters are delimited using an ampersand (&). Query strings may be used for passing additional data between individual application pages.

For example, a search result page written in PHP may use a URL like http://example.com/results.php?item=shirt&season=summer. In this example, two additional parameters are passed to the imaginary result.php application script: item, with the value shirt, and season with the value summer. The application may use the query string information to build the right page for the visitor.

Apache rewrite rules are often employed to simplify such long and unpleasant links as the example above into friendly URLs that are easier to type and interpret visually. In this example, we would like to simplify the above link to become http://example.com/shirt/summer. The shirt and summer parameter values are still in the address, but without the query string and script name.

Here's one rule to implement this:

Simple substition

RewriteRule ^shirt/summer$ results.php?item=shirt&season=summer [QSA]

The shirt/summer is explicitly matched in the requested address and Apache is told to serve results.php?item=shirt&season=summer instead.

The [QSA] flags are commonly used in rewrite rules. They tell Apache to append any additional query string to the served URL, so if the visitor types http://example.com/shirt/summer?page=2 the server will respond with results.php?item=shirt&season=summer&page=2. Without it, the additional query string would get discarded.

While this method achieves the desired effect, both the item name and season are hardcoded into the rule. This means the rule will not work for any other items, like pants, or seasons, like winter.

To make the rule more generic, we can use regular expressions to match parts of the original address and use those parts in a substitution pattern. The modified rule will then look like this:

Simple substition

RewriteRule ^([A-Za-z0-9]+)/(summer|winter|fall|spring) results.php?item=$1&season=$2 [QSA]

The first regular expression group in parenthesis matches a string containing alphanumeric characters and numbers like shirt or pants and saves the matched fragment as the $1 variable. The second regular expression group in parentheses matches exactly summer, winter, fall, or spring, and similarly saves the matched fragment as $2.

The matched fragments are then used in the resulting URL in item and season variables instead of the hardcoded shirt and summer values we used before.

The above will convert, for example, http://example.com/pants/summer into http://example.com/results.php?item=pants&season=summer. This example is also future proof, allowing multiple items and seasons to be correctly rewritten using a single rule.

Example 2 — Adding Conditions with Logic Using RewriteConds

Rewrite rules are not necessarily always evaluated one by one without any limitations. The RewriteCond directive lets us add conditions to our rewrite rules to control when the rules will be processed. All RewriteConds abide by the following format:

General RewriteCond structure

RewriteCond TestString Condition [Flags]

RewriteCond specifies the RewriteCond directive.
TestString is the string to test against.
Condition is the pattern or condition to match.
Flags are optional parameters that may modify the condition and evaluation rules.

If a RewriteCond evaluates to true, the next RewriteRule will be considered. If it doesn't, the rule will be discarded. Multiple RewriteConds may be used one after another, though all must evaluate to true for the next rule to be considered.

As an example, let's assume you would like to redirect all requests to non-existent files or directories on your site back to the home page instead of showing the standard 404 Not Found error page. This can be achieved with following conditions rules:

Redirect all requests to non-existent files and directories to home page

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /

With the above:

%{REQUEST_FILENAME} is the string to check. In this case, it's the requested filename, which is a system variable available for every request.
-f is a built-in condition which verifies if the requested name exists on disk and is a file. The ! is a negation operator. Combined, !-f evaluates to true only if a specified name does not exist or is not a file.
Similarly, !-d evaluates to true only if a specified name does not exist or is not a directory.

The RewriteRule on the final line will come into effect only for requests to non-existent files or directories. The RewriteRule itself is very simple and redirects every request to the / website root.

Conclusion

mod_rewrite lets you create human-readable URLs. In this tutorial, you learned how to use the RewriteRule directive to redirect URLs, including ones with query strings. You also learned how to conditionally redirect URLs using the RewriteCond directive.

If you'd like to learn more about mod_rewrite, take a look at Apache's mod_rewrite Introduction and Apache's official documentation for mod_rewrite.

How To Install and Use Composer on Debian 9

2018-09-07T16:09:10Z

Introduction

Composer is a popular dependency management tool for PHP, created mainly to facilitate installation and updates for project dependencies. It will check which other packages a specific project depends on and install them for you, using the appropriate versions according to the project requirements.

In this tutorial, you'll install and get started with Composer on Debian 9.

Prerequisites

To complete this tutorial, you will need:

One Debian 9 server set up by following the Debian 9 initial server setup guide, including a non-root user with sudo access and a firewall.

Step 1 — Installing the Dependencies

Before you download and install Composer, ensure your server has all dependencies installed.

First, update the package manager cache by running:

sudo apt update

Now, let's install the dependencies. We'll need curl in order to download Composer and php-cli for installing and running it. The php-mbstring package is necessary to provide functions for a library we'll be using. git is used by Composer for downloading project dependencies, and unzip for extracting zipped packages. Everything can be installed with the following command:

sudo apt install curl php-cli php-mbstring git unzip

With the prerequisites installed, we can install Composer itself.

Step 2 — Downloading and Installing Composer

Composer provides an installer, written in PHP. We'll download it, verify that it's not corrupted, and then use it to install Composer.

Make sure you're in your home directory, then retrieve the installer using curl:

cd ~
curl -sS https://getcomposer.org/installer -o composer-setup.php

Next, verify that the installer matches the SHA-384 hash for the latest installer found on the [Composer Public Keys / Signatures][composer-sigs] page. Copy the hash from that page and store it as a shell variable:

HASH=544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061

Make sure that you substitute the latest hash for the highlighted value.

Now execute the following PHP script to verify that the installation script is safe to run:

php -r "if (hash_file('SHA384', 'composer-setup.php') === '$HASH') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

You'll see the following output.

Output

Installer verified

If you see Installer corrupt, then you'll need to redownload the installation script again and double check that you're using the correct hash. Then run the command to verify the installer again. Once you have a verified installer, you can continue.

To install composer globally, use the following command which will download and install Composer as a system-wide command named composer, under /usr/local/bin:

sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer

You'll see the following output:

OutputAll settings correct for using Composer
Downloading...

Composer (version 1.7.2) successfully installed to: /usr/local/bin/composer
Use it: php /usr/local/bin/composer

To test your installation, run:

composer

And you'll see this output displaying Composer's version and arguments.

Output   ______
  / ____/___  ____ ___  ____  ____  ________  _____
 / /   / __ \/ __ `__ \/ __ \/ __ \/ ___/ _ \/ ___/
/ /___/ /_/ / / / / / / /_/ / /_/ (__  )  __/ /
\____/\____/_/ /_/ /_/ .___/\____/____/\___/_/
                    /_/
Composer version 1.7.2 2018-08-16 16:57:12

Usage:
  command [options] [arguments]

Options:
  -h, --help                     Display this help message
  -q, --quiet                    Do not output any message
  -V, --version                  Display this application version
      --ansi                     Force ANSI output
      --no-ansi                  Disable ANSI output
  -n, --no-interaction           Do not ask any interactive question
      --profile                  Display timing and memory usage information
      --no-plugins               Whether to disable plugins.
  -d, --working-dir=WORKING-DIR  If specified, use the given directory as working directory.
  -v|vv|vvv, --verbose           Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
. . .

This verifies that Composer installed successfully on your system and is available system-wide.

Note: If you prefer to have separate Composer executables for each project you host on this server, you can install it locally, on a per-project basis. Users of NPM will be familiar with this approach. This method is also useful when your system user doesn't have permission to install software system-wide.

To do this, use the command php composer-setup.php. This will generate a composer.phar file in your current directory, which can be executed with ./composer.phar command.

Now let's look at using Composer to manage dependencies.

Step 3 — Using Composer in a PHP Project

PHP projects often depend on external libraries, and managing those dependencies and their versions can be tricky. Composer solves that by tracking your dependencies and making it easy for others to install them.

In order to use Composer in your project, you'll need a composer.json file. The composer.json file tells Composer which dependencies it needs to download for your project, and which versions of each package are allowed to be installed. This is extremely important to keep your project consistent and avoid installing unstable versions that could potentially cause backwards compatibility issues.

You don't need to create this file manually - it's easy to run into syntax errors when you do so. Composer auto-generates the composer.json file when you add a dependency to your project using the require command. You can add additional dependencies in the same way, without the need to manually edit this file.

The process of using Composer to install a package as dependency in a project involves the following steps:

Identify what kind of library the application needs.
Research a suitable open source library on Packagist.org, the official package repository for Composer.
Choose the package you want to depend on.
Run composer require to include the dependency in the composer.json file and install the package.

Let's try this out with a demo application.

The goal of this application is to transform a given sentence into a URL-friendly string - a slug. This is commonly used to convert page titles to URL paths (like the final portion of the URL for this tutorial).

Let's start by creating a directory for our project. We'll call it slugify:

cd ~
mkdir slugify
cd slugify

Now it's time to search Packagist.org for a package that can help us generate slugs. If you search for the term "slug" on Packagist, you'll get a result similar to this:

You'll see two numbers on the right side of each package in the list. The number on the top represents how many times the package was installed, and the number on the bottom shows how many times a package was starred on GitHub. You can reorder the search results based on these numbers (look for the two icons on the right side of the search bar). Generally speaking, packages with more installations and more stars tend to be more stable, since so many people are using them. It's also important to check the package description for relevance to make sure it's what you need.

We need a simple string-to-slug converter. From the search results, the package cocur/slugify seems to be a good match, with a reasonable amount of installations and stars. (The package is a bit further down the page than the screenshot shows.)

Packages on Packagist have a vendor name and a package name. Each package has a unique identifier (a namespace) in the same format GitHub uses for its repositories, in the form vendor/package. The library we want to install uses the namespace cocur/slugif. You need the namespace in order to require the package in your project.

Now that you know exactly which package you want to install, run composer require to include it as a dependency and also generate the composer.json file for the project:

composer require cocur/slugify

You'll see this output as Composer downloads the dependency:

Output
Using version ^3.1 for cocur/slugify
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing cocur/slugify (v3.1): Downloading (100%)
Writing lock file
Generating autoload files

As you can see from the output, Composer automatically decided which version of the package to use. If you check your project's directory now, it will contain two new files: composer.json and composer.lock, and a vendor directory:

ls -l

Outputtotal 12
-rw-r--r-- 1 sammy sammy   59 Sep  7 16:03 composer.json
-rw-r--r-- 1 sammy sammy 2934 Sep  7 16:03 composer.lock
drwxr-xr-x 4 sammy sammy 4096 Sep  7 16:03 vendor

The composer.lock file is used to store information about which versions of each package are installed, and ensure the same versions are used if someone else clones your project and installs its dependencies. The vendor directory is where the project dependencies are located. The vendor folder doesn't need to be committed into version control - you only need to include the composer.json and composer.lock files.

When installing a project that already contains a composer.json file, run composer install in order to download the project's dependencies.

Let's take a quick look at version constraints. If you check the contents of your composer.json file, you'll see something like this:

cat composer.json

Output{
    "require": {
        "cocur/slugify": "^3.1"
    }
}

You might notice the special character ^ before the version number in composer.json. Composer supports several different constraints and formats for defining the required package version, in order to provide flexibility while also keeping your project stable. The caret (^) operator used by the auto-generated composer.json file is the recommended operator for maximum interoperability, following semantic versioning. In this case, it defines 3.1 as the minimum compatible version, and allows updates to any future version below 4.0.

Generally speaking, you won't need to tamper with version constraints in your composer.json file. However, some situations might require that you manually edit the constraints–for instance, when a major new version of your required library is released and you want to upgrade, or when the library you want to use doesn't follow semantic versioning.

Here are some examples to give you a better understanding of how Composer version constraints work:

Constraint	Meaning	Example Versions Allowed
^1.0	>= 1.0 < 2.0	1.0, 1.2.3, 1.9.9
^1.1.0	>= 1.1.0 < 2.0	1.1.0, 1.5.6, 1.9.9
~1.0	>= 1.0 < 2.0.0	1.0, 1.4.1, 1.9.9
~1.0.0	>= 1.0.0 < 1.1	1.0.0, 1.0.4, 1.0.9
1.2.1	1.2.1	1.2.1
1.*	>= 1.0 < 2.0	1.0.0, 1.4.5, 1.9.9
1.2.*	>= 1.2 < 1.3	1.2.0, 1.2.3, 1.2.9

For a more in-depth view of Composer version constraints, see the official documentation.

Next, let's look at how to load dependencies automatically with Composer.

Step 4 — Including the Autoload Script

Since PHP itself doesn't automatically load classes, Composer provides an autoload script that you can include in your project to get autoloading for free. This makes it much easier to work with your dependencies.

The only thing you need to do is include the vendor/autoload.php file in your PHP scripts before any class instantiation. This file is automatically generated by Composer when you add your first dependency.

Let's try it out in our application. Create the file test.php and open it in your text editor:

nano test.php

Add the following code which brings in the vendor/autoload.php file, loads the cocur/slugify dependency, and uses it to create a slug:

test.php

<?php require __DIR__ . '/vendor/autoload.php'; 
use Cocur\Slugify\Slugify;

$slugify = new Slugify();

echo $slugify->slugify('Hello World, this is a long sentence and I need to make a slug from it!');

Save the file and exit your editor.

Now run the script:

php test.php

This produces the output hello-world-this-is-a-long-sentence-and-i-need-to-make-a-slug-from-it.

Dependencies need updates when new versions come out, so let's look at how to handle that.

Step 5 — Updating Project Dependencies

Whenever you want to update your project dependencies to more recent versions, run the update command:

composer update

This will check for newer versions of the libraries you required in your project. If a newer version is found and it's compatible with the version constraint defined in the composer.json file, Composer will replace the previous version installed. The composer.lock file will be updated to reflect these changes.

You can also update one or more specific libraries by specifying them like this:

composer update vendor/package vendor2/package2

Be sure to check in your composer.json and composer.lock files after you update your dependencies so that others can install these newer versions.

Conclusion

Composer is a powerful tool every PHP developer should have in their utility belt. In this tutorial you installed Composer on Debian 9 and used it in a simple project. You now know how to install and update dependencies.

Beyond providing an easy and reliable way for managing project dependencies, it also establishes a new de facto standard for sharing and discovering PHP packages created by the community.

How To Install and Configure ownCloud on Debian 9

2018-09-07T15:58:15Z

Introduction

ownCloud is an open-source file sharing server and collaboration platform that can store your personal content, like documents and pictures, in a centralized location. This allows you to take control of your content and security by not relying on third-party content hosting services like Dropbox.

In this tutorial, we will install and configure an ownCloud instance on a Debian 9 server.

Prerequisites

In order to complete the steps in this guide, you will need the following:

A sudo user and firewall on your server: You can create a user with sudo privileges and set up a basic firewall by following the Debian 9 initial server setup guide.
A LAMP stack: ownCloud requires a web server, a database, and PHP to function properly. Setting up a LAMP stack (Linux, Apache, MySQL, and PHP) server fulfills all of these requirements. Follow this guide to install and configure this software.
An SSL certificate: How you set this up depends on whether or not you have a domain name that resolves to your server.
- If you have a domain name... the easiest way to secure your site is with Let's Encrypt, which provides free, trusted certificates. Follow the Let's Encrypt guide for Apache to set this up.
- If you do not have a domain... and you are just using this configuration for testing or personal use, you can use a self-signed certificate instead. This provides the same type of encryption, but without the domain validation. Follow the self-signed SSL guide for Apache to get set up.

Step 1 – Installing ownCloud

The ownCloud server package does not exist within the default repositories for Debian. However, ownCloud maintains a dedicated repository for the distribution that we can add to our server.

To begin, let's install a few components to help us add the ownCloud repositories. The apt-transport-https package allows us to use the deb https:// in our apt sources list to indicate external repositories served over HTTPS:

sudo apt update
sudo apt install curl apt-transport-https

Next, download the ownCloud release key using the curl command and import it with the apt-key utility with the add command:

curl https://download.owncloud.org/download/repositories/production/Debian_9.0/Release.key | sudo apt-key add -

The 'Release.key' file contains a PGP (Pretty Good Privacy) public key which apt will use to verify that the ownCloud package is authentic.

In addition to importing the key, create a file called owncloud.list in the sources.list.d directory for apt. The file will contain the address to the ownCloud repository.

echo 'deb http://download.owncloud.org/download/repositories/production/Debian_9.0/ /' | sudo tee /etc/apt/sources.list.d/owncloud.list

Now, we can use the package manager to find and install ownCloud. Along with the main package, we will also install a few additional PHP libraries that ownCloud uses to add extra functionality. Update your local package index and install everything by typing:

sudo apt update
sudo apt install php-bz2 php-curl php-gd php-imagick php-intl php-mbstring php-xml php-zip owncloud-files

Everything we need is now installed on the server, so next we can finish the configuration and we can begin using the service.

Step 2 — Adjusting the Document Root

The ownCloud package we installed copies the web files to /var/www/owncloud on the server. Currently, the Apache virtual host configuration is set up to serve files out of a different directory. We need to change the DocumentRoot setting in our configuration to point to the new directory.

You find which virtual host files reference your domain name or IP address using the apache2ctl utility with the DUMP_VHOSTS option. Filter the output by your server's domain name or IP address to find which files you need to edit in the next few commands:

sudo apache2ctl -t -D DUMP_VHOSTS | grep server_domain_or_IP

The output will probably look something like this:

Output
*:443                  server_domain_or_IP (/etc/apache2/sites-enabled/server_domain_or_IP-le-ssl.conf:2)
         port 80 namevhost server_domain_or_IP (/etc/apache2/sites-enabled/server_domain_or_IP.conf:1)

In the parentheses, you can see each of the files that reference the domain name or IP address we'll use to access ownCloud. These are the files you'll need to edit.

For each match, open the file in a text editor with sudo privileges:

sudo nano /etc/apache2/sites-enabled/server_domain_or_IP.conf

Inside, search for the DocumentRoot directive. Change the line so that it points to the /var/www/owncloud directory:

Example DocumentRoot edit

<VirtualHost *:80>
    . . .
    DocumentRoot /var/www/owncloud
    . . .
</VirtualHost>

Save and close the file when you are finished. Complete this process for each of the files that referenced your domain name (or IP address if you did not configure a domain for your server).

When you are finished, check the syntax of your Apache files to make sure there were no detectable typos in your configuration:

sudo apache2ctl configtest

OutputSyntax OK

Depending on your configuration, you may see a warning about setting ServerName globally. As long as the output ends with Syntax OK, you can ignore that warning. If you see additional errors, go back and check the files you just edited for mistakes.

If your syntax check passed, reload the Apache service to activate the new changes:

sudo systemctl reload apache2

Apache should now know how to server your ownCloud files.

Step 3 – Configuring the MySQL Database

Before we move on to the web configuration, we need to set up the database. During the web-based configuration process, we will need to provide an database name, a database username, and a database password so that ownCloud can connect and manage its information within MySQL.

Begin by logging into your database with the MySQL administrative account:

sudo mysql

If you set up password authentication for a MySQL administrative account, you may have to use this syntax instead:

mysql -u admin -p

Create a dedicated database for ownCloud to use. We will name the database owncloud for clarity:

CREATE DATABASE owncloud;

Note: Every MySQL statement must end with a semi-colon (;). Be sure to check that this is present if you are experiencing an issue.

Next, create a separate MySQL user account to manage the newly created database. Creating one-function databases and accounts is a good idea from a management and security standpoint. As with the naming of the database, choose a username that you prefer. We elected to go with the name owncloud in this guide.

GRANT ALL ON owncloud.* to 'owncloud'@'localhost' IDENTIFIED BY 'owncloud_database_password';

Warning: Be sure to put an actual password where the command states: owncloud_database_password

With the user assigned access to the database, perform the flush privileges operation to ensure that the running instance of MySQL knows about the recent privilege assignment:

FLUSH PRIVILEGES;

You can now exit the MySQL session by typing:

exit

With the ownCloud server installed and the database set up, we are ready to turn our attention to configuring the ownCloud application.

Step 4 – Configuring ownCloud

To access the ownCloud web interface, open a web browser and navigate to the following address:

https://server_domain_or_IP

Note: If you are using a self-signed SSL certificate, you will likely be presented with a warning because the certificate is not signed by one of your browser's trusted authorities. This is expected and normal. Click the appropriate button or link to proceed to the ownCloud admin page.

You should see the ownCloud web configuration page in your browser.

Create an admin account by choosing a username and a password. For security purposes it is not recommended to use something like "admin" for the username:

Next, leave the Data folder setting as-is and scroll down to the database configuration section.

Fill out the details of the database name, database username, and database password you created in the previous section. If you used the settings from this guide, both the database name and username will be owncloud. Leave the database host as localhost:

Click the Finish setup button to finish configuring ownCloud using the information you've provided. You will be taken to a login screen where you can sign in using your new account:

On your first login, a screen will appear where you can download applications to sync your files on various devices. You can download and configure these now or do it at a later time. When you are finished, click the x in the top-right corner of the splash screen to access the main interface:

Here, you can create or upload files to your personal cloud.

Conclusion

ownCloud can replicate the capabilities of popular third-party cloud storage services. Content can be shared between users or externally with public URLs. The advantage of ownCloud is that the information is stored in a place that you control and manage without a third party.

Explore the interface and for additional functionality, install plugins using ownCloud's app store.

How To Install WordPress with LAMP on Debian 9

2018-09-07T15:07:57Z

Introduction

WordPress is the most popular CMS (content management system) on the internet. It allows you to easily set up flexible blogs and websites on top of a MariaDB backend with PHP processing. WordPress has seen incredible adoption and is a great choice for getting a website up and running quickly. After setup, almost all administration can be done through the web frontend.

In this guide, we'll focus on getting a WordPress instance set up on a LAMP stack (Linux, Apache, MariaDB, and PHP) on a Debian 9 server.

Prerequisites

In order to complete this tutorial, you will need access to a Debian 9 server.

You will need to perform the following tasks before you can start this guide:

Create a sudo user on your server: We will be completing the steps in this guide using a non-root user with sudo privileges. You can create a user with sudo privileges by following our Debian 9 initial server setup guide.
Install a LAMP stack: WordPress will need a web server, a database, and PHP in order to correctly function. Setting up a LAMP stack (Linux, Apache, MariaDB, and PHP) fulfills all of these requirements. Follow this guide to install and configure this software.
Secure your site with SSL: WordPress serves dynamic content and handles user authentication and authorization. TLS/SSL is the technology that allows you to encrypt the traffic from your site so that your connection is secure. The way you set up SSL will depend on whether you have a domain name for your site.
- If you have a domain name... the easiest way to secure your site is with Let's Encrypt, which provides free, trusted certificates. Follow our Let's Encrypt guide for Apache to set this up.
- If you do not have a domain... and you are just using this configuration for testing or personal use, you can use a self-signed certificate instead. This provides the same type of encryption, but without the domain validation. Follow our self-signed SSL guide for Apache to get set up.

When you are finished with the setup steps, log in to your server as your sudo user and continue below.

Step 1 — Creating a MariaDB Database and User for WordPress

The first step that we will take is a preparatory one. WordPress uses MySQL to manage and store site and user information. We have MariaDB — a drop-in replacement for MySQL — installed already, but we need to make a database and a user for WordPress to use.

To get started, open up the MariaDB prompt as the root account:

sudo mariadb

Note: If you set up another account with administrative privileges when you installed and set up MariaDB, you can also log in as that user. You’ll need to do so with the following command:

mariadb -u username -p

After issuing this command, MariaDB will prompt you for the password you set for that account.

Begin by creating a new database that WordPress will control. You can call this whatever you would like but, to keep it simple for this guide, we will name it wordpress.

Create the database for WordPress by typing:

CREATE DATABASE wordpress DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Note that every MySQL statement must end in a semi-colon (;). Check to make sure this is present if you are running into any issues.

Next, create a separate MySQL user account that we will use exclusively to operate on our new database. Creating single-function databases and accounts is a good idea from a management and security standpoint. We will use the name wordpressuser in this guide, but feel free to change this if you'd like.

Create this account, set a password, and grant the user access to the database you just created with the following command. Remember to choose a strong password for your database user:

GRANT ALL ON wordpress.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password';

You now have a database and user account, each made specifically for WordPress. Run the following command to reload the grant tables so that the current instance of MariaDB knows about the changes you've made:

FLUSH PRIVILEGES;

Exit out of MariaDB by typing:

EXIT;

Now that you’ve configured the database and user that will be used by WordPress, you can move on to installing some PHP-related packages used by the CMS.

Step 2 — Installing Additional PHP Extensions

When setting up our LAMP stack, we only required a very minimal set of extensions in order to get PHP to communicate with MariaDB. WordPress and many of its plugins leverage additional PHP extensions.

Download and install some of the most popular PHP extensions for use with WordPress by typing:

sudo apt update
sudo apt install php-curl php-gd php-mbstring php-xml php-xmlrpc php-soap php-intl php-zip

Note: Each WordPress plugin has its own set of requirements. Some may require additional PHP packages to be installed. Check your plugin documentation to find its PHP requirements. If they are available, they can be installed with apt as demonstrated above.

We will restart Apache to load these new extensions in the next section. If you are returning here to install additional plugins, you can restart Apache now by typing:

sudo systemctl restart apache2

At this point, all that’s left to do before installing WordPress is to make some changes to your Apache configuration in order to allow the CMS to function smoothly.

Step 3 — Adjusting Apache's Configuration to Allow for .htaccess Overrides and Rewrites

With the additional PHP extensions installed and ready for use, the next thing to do is to make a few changes to your Apache configuration. Based on the prerequisite tutorials, you should have a configuration file for your site in the /etc/apache2/sites-available/ directory. We'll use /etc/apache2/sites-available/wordpress.conf as an example here, but you should substitute the path to your configuration file where appropriate.

Additionally, we will use /var/www/wordpress as the root directory of our WordPress install. You should use the web root specified in your own configuration.

Note: It's possible you are using the 000-default.conf default configuration (with /var/www/html as your web root). This is fine to use if you're only going to host one website on this server. If not, it's best to split the necessary configuration into logical chunks, one file per site.

Currently, the use of .htaccess files is disabled. WordPress and many WordPress plugins use these files extensively for in-directory tweaks to the web server's behavior.

Open the Apache configuration file for your website. Note that if you have an existing Apache configuration file for your website, this file’s name will be different:

sudo nano /etc/apache2/sites-available/wordpress.conf

To allow .htaccess files, you’ll need to add a Directory block pointing to your document root with an AllowOverride directive within it. Add the following block of text inside the VirtualHost block in your configuration file, being sure to use the correct web root directory:

/etc/apache2/sites-available/wordpress.conf

<Directory /var/www/wordpress/>
    AllowOverride All
</Directory>

When you are finished, save and close the file.

Next, enable the rewrite module in order to utilize the WordPress permalink feature:

sudo a2enmod rewrite

Before implementing the changes you've made, check to make sure that you haven't made any syntax errors:

sudo apache2ctl configtest

If your configuration file’s syntax is correct, you’ll see the following in your output:

OutputSyntax OK

If this command reports any errors, go back and check that you haven’t made any syntax errors in your configuration file. Otherwise, restart Apache to implement the changes:

sudo systemctl restart apache2

Next, we will download and set up WordPress itself.

Step 4 — Downloading WordPress

Now that your server software is configured, you can download and set up WordPress. For security reasons in particular, it is always recommended to get the latest version of WordPress directly from their site.

Note: We will use curl to download WordPress, but this program may not be installed by default on your Debian server. To install it, run:

sudo apt install curl

Change into a writable directory and then download the compressed release by typing:

cd /tmp
curl -O https://wordpress.org/latest.tar.gz

Extract the compressed file to create the WordPress directory structure:

tar xzvf latest.tar.gz

We will move these files into our document root momentarily. Before we do, though, add a dummy .htaccess file so that this will be available for WordPress to use later.

Create the file by typing:

touch /tmp/wordpress/.htaccess

Then copy over the sample configuration file to the filename that WordPress actually reads:

cp /tmp/wordpress/wp-config-sample.php /tmp/wordpress/wp-config.php

Additionally, create the upgrade directory so that WordPress won't run into permissions issues when trying to do this on its own following an update to its software:

mkdir /tmp/wordpress/wp-content/upgrade

Then, copy the entire contents of the directory into your document root. Notice that the following command includes a dot at the end of the source directory to indicate that everything within the directory should be copied, including hidden files (like the .htaccess file you created):

sudo cp -a /tmp/wordpress/. /var/www/wordpress

With that, you’ve successfully installed WordPress onto your web server and performed some of the initial configuration steps. Next, we’ll discuss some further configuration changes that will give WordPress the privileges it needs to function as well as access to the MariaDB database and user account you created previously.

Step 5 — Configuring the WordPress Directory

Before we can go through the web-based setup process for WordPress, we need to adjust some items in our WordPress directory.

Start by giving ownership of all the files to the www-data user and group. This is the user that the Apache web server runs as, and Apache will need to be able to read and write WordPress files in order to serve the website and perform automatic updates.

Update the ownership with chown:

sudo chown -R www-data:www-data /var/www/wordpress

Next we will run two find commands to set the correct permissions on the WordPress directories and files:

sudo find /var/www/wordpress/ -type d -exec chmod 750 {} \;
sudo find /var/www/wordpress/ -type f -exec chmod 640 {} \;

These should be a reasonable permissions set to start with, although some plugins and procedures might require additional tweaks.

Following this, you will need to make some changes to the main WordPress configuration file.

When you open the file, your first objective will be to adjust some secret keys to provide some security for your installation. WordPress provides a secure generator for these values so that you do not have to try to come up with good values on your own. These are only used internally, so it won't hurt usability to have complex, secure values here.

To grab secure values from the WordPress secret key generator, type:

curl -s https://api.wordpress.org/secret-key/1.1/salt/

You will get back unique values that look something like this:

Warning! It is important that you request unique values each time. Do NOT copy the values shown below!

Output
define('AUTH_KEY',         '1jl/vqfs<XhdXoAPz9 DO NOT COPY THESE VALUES c_j{iwqD^<+c9.k<J@4H');
define('SECURE_AUTH_KEY',  'E2N-h2]Dcvp+aS/p7X DO NOT COPY THESE VALUES {Ka(f;rv?Pxf})CgLi-3');
define('LOGGED_IN_KEY',    'W(50,{W^,OPB%PB<JF DO NOT COPY THESE VALUES 2;y&,2m%3]R6DUth[;88');
define('NONCE_KEY',        'll,4UC)7ua+8<!4VM+ DO NOT COPY THESE VALUES #`DXF+[$atzM7 o^-C7g');
define('AUTH_SALT',        'koMrurzOA+|L_lG}kf DO NOT COPY THESE VALUES  07VC*Lj*lD&?3w!BT#-');
define('SECURE_AUTH_SALT', 'p32*p,]z%LZ+pAu:VY DO NOT COPY THESE VALUES C-?y+K0DK_+F|0h{!_xY');
define('LOGGED_IN_SALT',   'i^/G2W7!-1H2OQ+t$3 DO NOT COPY THESE VALUES t6**bRVFSD[Hi])-qS`|');
define('NONCE_SALT',       'Q6]U:K?j4L%Z]}h^q7 DO NOT COPY THESE VALUES 1% ^qUswWgn+6&xqHN&%');

These are configuration lines that you will paste directly into your configuration file to set secure keys. Copy the output you received to your clipboard, and then open the WordPress configuration file located in your document root:

sudo nano /var/www/wordpress/wp-config.php

Find the section that contains the dummy values for those settings. It will look something like this:

/var/www/wordpress/wp-config.php

. . .

define('AUTH_KEY',         'put your unique phrase here');
define('SECURE_AUTH_KEY',  'put your unique phrase here');
define('LOGGED_IN_KEY',    'put your unique phrase here');
define('NONCE_KEY',        'put your unique phrase here');
define('AUTH_SALT',        'put your unique phrase here');
define('SECURE_AUTH_SALT', 'put your unique phrase here');
define('LOGGED_IN_SALT',   'put your unique phrase here');
define('NONCE_SALT',       'put your unique phrase here');

. . .

Delete these lines and paste in the values you copied from the command line:

/var/www/wordpress/wp-config.php

. . .

define('AUTH_KEY',         'VALUES COPIED FROM THE COMMAND LINE');
define('SECURE_AUTH_KEY',  'VALUES COPIED FROM THE COMMAND LINE');
define('LOGGED_IN_KEY',    'VALUES COPIED FROM THE COMMAND LINE');
define('NONCE_KEY',        'VALUES COPIED FROM THE COMMAND LINE');
define('AUTH_SALT',        'VALUES COPIED FROM THE COMMAND LINE');
define('SECURE_AUTH_SALT', 'VALUES COPIED FROM THE COMMAND LINE');
define('LOGGED_IN_SALT',   'VALUES COPIED FROM THE COMMAND LINE');
define('NONCE_SALT',       'VALUES COPIED FROM THE COMMAND LINE');

. . .

Next, modify the database connection settings at the top of the file. You need to adjust the database name, the database user, and the associated password that you’ve configured within MariaDB.

The other change you must make is to set the method that WordPress should use to write to the filesystem. Since we've given the web server permission to write where it needs to, we can explicitly set the filesystem method to "direct". Failure to set this with our current settings would result in WordPress prompting for FTP credentials when you perform certain actions.

This setting can be added below the database connection settings, or anywhere else in the file:

/var/www/wordpress/wp-config.php

. . .

define('DB_NAME', 'wordpress');

/** MySQL database username */
define('DB_USER', 'wordpressuser');

/** MySQL database password */
define('DB_PASSWORD', 'password');

. . .

define('FS_METHOD', 'direct');

Save and close the file when you are finished. Finally, you can finish installing and configuring WordPress by accessing it through your web browser.

Step 6 — Completing the Installation Through the Web Interface

Now that the server configuration is complete, we can complete the installation through the web interface.

In your web browser, navigate to your server's domain name or public IP address:

https://server_domain_or_IP

Select the language you would like to use:

Next, you will come to the main setup page. Select a name for your WordPress site and choose a username (it is recommended not to choose something like "admin" for security purposes). A strong password is generated automatically. Save this password or select an alternative strong password.

Enter your email address and select whether you want to discourage search engines from indexing your site:

When ready, click the Install WordPress button. You’ll be taken to a page that prompts you to log in:

Once you log in, you will be taken to the WordPress administration dashboard:

From the dashboard, you can begin making changes to your site’s theme and publishing content.

Conclusion

WordPress should be installed and ready to use! Some common next steps are to choose the permalinks setting for your posts (which can be found in Settings > Permalinks) or to select a new theme (in Appearance > Themes). If this is your first time using WordPress, explore the interface a bit to get acquainted with your new CMS.

How To Create a Self-Signed SSL Certificate for Apache in Debian 9

2018-09-07T14:37:06Z

Introduction

TLS, or transport layer security, and its predecessor SSL, which stands for secure sockets layer, are web protocols used to wrap normal traffic in a protected, encrypted wrapper.

Using this technology, servers can send traffic safely between servers and clients without the possibility of messages being intercepted by outside parties. The certificate system also assists users in verifying the identity of the sites that they are connecting with.

In this guide, we will show you how to set up a self-signed SSL certificate for use with an Apache web server on Debian 9.

A self-signed certificate may be appropriate if you do not have a domain name associated with your server and for instances where an encrypted web interface is not user-facing. If you do have a domain name, in many cases it is better to use a CA-signed certificate. You can find out how to set up a free trusted certificate with the Let's Encrypt project here.

Prerequisites

Before you begin, you should have a non-root user configured with sudo privileges. You can learn how to set up such a user account by following our Initial Server Setup with Debian 9.

You will also need to have the Apache web server installed. If you would like to install an entire LAMP (Linux, Apache, MariaDB, PHP) stack on your server, you can follow our guide on setting up LAMP on Debian 9. If you just want the Apache web server, skip the steps pertaining to PHP and MariaDB.

When you have completed these prerequisites, continue below.

Step 1 — Creating the SSL Certificate

We can create a self-signed key and certificate pair with OpenSSL in a single command:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt

You will be asked a series of questions. Before we go over that, let's take a look at what is happening in the command we are issuing:

openssl: This is the basic command line tool for creating and managing OpenSSL certificates, keys, and other files.
req: This subcommand specifies that we want to use X.509 certificate signing request (CSR) management. The "X.509" is a public key infrastructure standard that SSL and TLS adheres to for its key and certificate management. We want to create a new X.509 cert, so we are using this subcommand.
-x509: This further modifies the previous subcommand by telling the utility that we want to make a self-signed certificate instead of generating a certificate signing request, as would normally happen.
-nodes: This tells OpenSSL to skip the option to secure our certificate with a passphrase. We need Apache to be able to read the file, without user intervention, when the server starts up. A passphrase would prevent this from happening because we would have to enter it after every restart.
-days 365: This option sets the length of time that the certificate will be considered valid. We set it for one year here.
-newkey rsa:2048: This specifies that we want to generate a new certificate and a new key at the same time. We did not create the key that is required to sign the certificate in a previous step, so we need to create it along with the certificate. The rsa:2048 portion tells it to make an RSA key that is 2048 bits long.
-keyout: This line tells OpenSSL where to place the generated private key file that we are creating.
-out: This tells OpenSSL where to place the certificate that we are creating.

As we stated above, these options will create both a key file and a certificate. We will be asked a few questions about our server in order to embed the information correctly in the certificate.

The entirety of the prompts will look something like this:

Output
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Bouncy Castles, Inc.
Organizational Unit Name (eg, section) []:Ministry of Water Slides
Common Name (e.g. server FQDN or YOUR name) []:server_IP_address
Email Address []:admin@your_domain.com

Both of the files you created will be placed in the appropriate subdirectories under /etc/ssl.

Step 2 — Configuring Apache to Use SSL

We have created our key and certificate files under the /etc/ssl directory. Now we just need to modify our Apache configuration to take advantage of these.

We will make a few adjustments to our configuration:

We will create a configuration snippet to specify strong default SSL settings.
We will modify the included SSL Apache Virtual Host file to point to our generated SSL certificates.
(Recommended) We will modify the unencrypted Virtual Host file to automatically redirect requests to the encrypted Virtual Host.

When we are finished, we should have a secure SSL configuration.

Creating an Apache Configuration Snippet with Strong Encryption Settings

First, we will create an Apache configuration snippet to define some SSL settings. This will set Apache up with a strong SSL cipher suite and enable some advanced features that will help keep our server secure. The parameters we will set can be used by any Virtual Hosts enabling SSL.

Create a new snippet in the /etc/apache2/conf-available directory. We will name the file ssl-params.conf to make its purpose clear:

sudo nano /etc/apache2/conf-available/ssl-params.conf

To set up Apache SSL securely, we will be using the recommendations by Remy van Elst on the Cipherli.st site. This site is designed to provide easy-to-consume encryption settings for popular software.

The choice of which config you use will depend largely on what you need to support. They both will provide great security.

For our purposes, we can copy the provided settings in their entirety. We will just make one small change to this and disable the Strict-Transport-Security header (HSTS).

Preloading HSTS provides increased security, but can have far-reaching consequences if accidentally enabled or enabled incorrectly. In this guide, we will not enable the settings, but you can modify that if you are sure you understand the implications.

Before deciding, take a moment to read up on HTTP Strict Transport Security, or HSTS, and specifically about the "preload" functionality.

Paste the following configuration into the ssl-params.conf file we opened:

/etc/apache2/conf-available/ssl-params.conf

SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
# Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
# Requires Apache >= 2.4.11
SSLSessionTickets Off

Save and close the file when you are finished.

Modifying the Default Apache SSL Virtual Host File

Next, let's modify /etc/apache2/sites-available/default-ssl.conf, the default Apache SSL Virtual Host file. If you are using a different server block file, substitute its name in the commands below.

Before we go any further, let's back up the original SSL Virtual Host file:

sudo cp /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-available/default-ssl.conf.bak

Now, open the SSL Virtual Host file to make adjustments:

sudo nano /etc/apache2/sites-available/default-ssl.conf

Inside, with most of the comments removed, the Virtual Host block should look something like this by default:

/etc/apache2/sites-available/default-ssl.conf

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin webmaster@localhost

                DocumentRoot /var/www/html

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on

                SSLCertificateFile      /etc/ssl/certs/ssl-cert-snakeoil.pem
                SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>

        </VirtualHost>
</IfModule>

We will be making some minor adjustments to the file. We will set the normal things we'd want to adjust in a Virtual Host file (ServerAdmin email address, ServerName, etc.), and adjust the SSL directives to point to our certificate and key files. Again, if you’re using a different document root, be sure to update the DocumentRoot directive.

After making these changes, your server block should look similar to this:

/etc/apache2/sites-available/default-ssl.conf

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin your_email@example.com
                ServerName server_domain_or_IP

                DocumentRoot /var/www/html

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on

                SSLCertificateFile      /etc/ssl/certs/apache-selfsigned.crt
                SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key

                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>

        </VirtualHost>
</IfModule>

Save and close the file when you are finished.

(Recommended) Modifying the HTTP Host File to Redirect to HTTPS

As it stands now, the server will provide both unencrypted HTTP and encrypted HTTPS traffic. For better security, it is recommended in most cases to redirect HTTP to HTTPS automatically. If you do not want or need this functionality, you can safely skip this section.

To adjust the unencrypted Virtual Host file to redirect all traffic to be SSL encrypted, open the /etc/apache2/sites-available/000-default.conf file:

sudo nano /etc/apache2/sites-available/000-default.conf

Inside, within the VirtualHost configuration blocks, add a Redirect directive, pointing all traffic to the SSL version of the site:

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
        . . .

        Redirect "/" "https://your_domain_or_IP/"

        . . .
</VirtualHost>

Save and close the file when you are finished.

That’s all of the configuration changes you need to make to Apache. Next, we will discuss how to update firewall rules with ufw to allow encrypted HTTPS traffic to your server.

Step 3 — Adjusting the Firewall

If you have the ufw firewall enabled, as recommended by the prerequisite guides, you might need to adjust the settings to allow for SSL traffic. Fortunately, when installed on Debian 9, ufw comes loaded with app profiles which you can use to tweak your firewall settings

We can see the available profiles by typing:

sudo ufw app list

You should see a list like this, with the following four profiles near the bottom of the output:

OutputAvailable applications:
. . .
  WWW
  WWW Cache
  WWW Full
  WWW Secure
. . .

You can see the current setting by typing:

sudo ufw status

If you allowed only regular HTTP traffic earlier, your output might look like this:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
WWW                        ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
WWW (v6)                   ALLOW       Anywhere (v6)

To additionally let in HTTPS traffic, allow the "WWW Full" profile and then delete the redundant "WWW" profile allowance:

sudo ufw allow 'WWW Full'
sudo ufw delete allow 'WWW'

Your status should look like this now:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
WWW Full                   ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
WWW Full (v6)              ALLOW       Anywhere (v6)

With your firewall configured to allow HTTPS traffic, you can move on to the next step where we’ll go over how to enable a few modules and configuration files to allow SSL to function properly.

Step 4 — Enabling the Changes in Apache

Now that we've made our changes and adjusted our firewall, we can enable the SSL and headers modules in Apache, enable our SSL-ready Virtual Host, and then restart Apache to put these changes into effect.

Enable mod_ssl, the Apache SSL module, and mod_headers, which is needed by some of the settings in our SSL snippet, with the a2enmod command:

sudo a2enmod ssl
sudo a2enmod headers

Next, enable your SSL Virtual Host with the a2ensite command:

sudo a2ensite default-ssl

You will also need to enable your ssl-params.conf file, to read in the values you’ve set:

sudo a2enconf ssl-params

At this point, the site and the necessary modules are enabled. We should check to make sure that there are no syntax errors in our files. Do this by typing:

sudo apache2ctl configtest

If everything is successful, you will get a result that looks like this:

OutputSyntax OK

As long as your output has Syntax OK in it, then your configuration file has no syntax errors and you can safely restart Apache to implement the changes:

sudo systemctl restart apache2

With that, your self-signed SSL certificate is all set. You can now test that your server is correctly encrypting its traffic.

Step 5 — Testing Encryption

You’re now ready to test your SSL server.

Open your web browser and type https:// followed by your server's domain name or IP into the address bar:

https://server_domain_or_IP

Because the certificate you created isn't signed by one of your browser's trusted certificate authorities, you will likely see a scary looking warning like the one below:

This is expected and normal. We are only interested in the encryption aspect of our certificate, not the third party validation of our host's authenticity. Click ADVANCED and then the link provided to proceed to your host anyways:

You should be taken to your site. If you look in the browser address bar, you will see a lock with an "x" over it or another similar “not secure” notice. In this case, this just means that the certificate cannot be validated. It is still encrypting your connection.

If you configured Apache to redirect HTTP to HTTPS, you can also check whether the redirect functions correctly:

http://server_domain_or_IP

If this results in the same icon, this means that your redirect worked correctly. However, the redirect you created earlier is only a temporary redirect. If you’d like to make the redirection to HTTPS permanent, continue on to the final step.

Step 6 — Changing to a Permanent Redirect

If your redirect worked correctly and you are sure you want to allow only encrypted traffic, you should modify the unencrypted Apache Virtual Host again to make the redirect permanent.

Open your server block configuration file again:

sudo nano /etc/apache2/sites-available/000-default.conf

Find the Redirect line we added earlier. Add permanent to that line, which changes the redirect from a 302 temporary redirect to a 301 permanent redirect:

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
        . . .

        Redirect permanent "/" "https://your_domain_or_IP/"

        . . .
</VirtualHost>

Save and close the file.

Check your configuration for syntax errors:

sudo apache2ctl configtest

If this command doesn’t report any syntax errors, restart Apache:

sudo systemctl restart apache2

This will make the redirect permanent, and your site will only serve traffic over HTTPS.

Conclusion

You have configured your Apache server to use strong encryption for client connections. This will allow you serve requests securely, and will prevent outside parties from reading your traffic.

How To Install and Configure GitLab on Debian 9

2018-09-07T15:04:08Z

Introduction

GitLab CE, or Community Edition, is an open-source application primarily used to host Git repositories, with additional development-related features like issue tracking. It is designed to be hosted using your own infrastructure, and provides flexibility in deploying as an internal repository store for your development team, a public way to interface with users, or a means for contributors to host their own projects.

The GitLab project makes it relatively straightforward to set up a GitLab instance on your own hardware with an easy installation mechanism. In this guide, we will cover how to install and configure GitLab on a Debian 9 server.

Prerequisites

For this tutorial, you will need:

A Debian 9 server with a non-root sudo user and basic firewall. To set this up, follow our Debian 9 initial server setup guide.

The published GitLab hardware requirements recommend using a server with:

2 cores
8GB of RAM

Although you may be able to get by with substituting some swap space for RAM, it is not recommended. For this guide we will assume that you have the above resources as a minimum.

A domain name pointed at your server. For more information, see our documentation on how to get started with DNS on DigitalOcean. This tutorial will use the domain name example.com for demonstration purposes.

Step 1 — Installing the Dependencies

Before we can install GitLab itself, it is important to install some of the software that it leverages during installation and on an ongoing basis. Fortunately, all of the required software can be easily installed from Debian's default package repositories.

Since this is our first time using apt during this session, we can refresh the local package index and then install the dependencies by typing:

sudo apt update
sudo apt install ca-certificates curl openssh-server postfix

You may have some of this software installed already. For the postfix installation, select Internet Site when prompted. On the next screen, enter your server's domain name to configure how the system will send mail.

Step 2 — Installing GitLab

Now that the dependencies are in place, we can install GitLab itself. This is a straightforward process that leverages an installation script to configure your system with the GitLab repositories.

Move into the /tmp directory and then download the installation script:

cd /tmp
curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh

Feel free to examine the downloaded script to ensure that you are comfortable with the actions it will take. You can also find a hosted version of the script here:

less /tmp/script.deb.sh

Once you are satisfied with the safety of the script, run the installer:

sudo bash /tmp/script.deb.sh

The script will set up your server to use the GitLab maintained repositories. This lets you manage GitLab with the same package management tools you use for your other system packages. Once this is complete, you can install the actual GitLab application with apt:

sudo apt install gitlab-ce

This will install the necessary components on your system.

Step 3 — Adjusting the Firewall Rules

Before you configure GitLab, you will need to ensure that your firewall rules are permissive enough to allow web traffic. If you followed the guide linked in the prerequisites, you will have a ufw firewall enabled.

View the current status of your active firewall by typing:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)

As you can see, the current rules allow SSH traffic through, but access to other services is restricted. Since GitLab is a web application, we should allow HTTP access. Because we will be taking advantage of GitLab's ability to request and enable a free TLS/SSL certificate from Let's Encrypt, let's also allow HTTPS access.

We can allow access to both HTTP and HTTPS by allowing the "WWW Full" app profile through our firewall. If you didn't already have OpenSSH traffic enabled, you should allow that traffic now too:

sudo ufw allow "WWW Full"
sudo ufw allow OpenSSH

Check the ufw status again, this time appending the verbose flag; you should see access configured to at least these two services:

sudo ufw status verbose

OutputStatus: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
80,443/tcp (WWW Full)      ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
80,443/tcp (WWW Full (v6)) ALLOW IN    Anywhere (v6)

The above output indicates that the GitLab web interface will be accessible once we configure the application.

Step 4 — Editing the GitLab Configuration File

Before you can use the application, you need to update the configuration file and run a reconfiguration command. First, open Gitlab's configuration file:

sudo nano /etc/gitlab/gitlab.rb

Near the top is the external_url configuration line. Update it to match your domain. Change http to https so that GitLab will automatically redirect users to the site protected by the Let's Encrypt certificate:

/etc/gitlab/gitlab.rb

##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'https://example.com'

Next, look for the letsencrypt['contact_emails'] setting. This setting defines a list of email addresses that the Let's Encrypt project can use to contact you if there are problems with your domain. It's a good idea to uncomment and fill this out so that you will know of any issues:

/etc/gitlab/gitlab.rb

letsencrypt['contact_emails'] = ['sammy@example.com']

Save and close the file. Run the following command to reconfigure Gitlab:

sudo gitlab-ctl reconfigure

This will initialize GitLab using the information it can find about your server. This is a completely automated process, so you will not have to answer any prompts. The process will also configure a Let's Encrypt certificate for your domain.

Step 5 — Performing Initial Configuration Through the Web Interface

With GitLab running and access permitted, we can perform some initial configuration of the application through the web interface.

Logging In for the First Time

Visit the domain name of your GitLab server in your web browser:

https://example.com

On your first time visiting, you should see an initial prompt to set a password for the administrative account:

In the initial password prompt, supply and confirm a secure password for the administrative account. Click on the Change your password button when you are finished.

You will be redirected to the conventional GitLab login page:

Here, you can log in with the password you just set. The credentials are:

Username: root
Password: [the password you set]

Enter these values into the fields for existing users and click the Sign in button. You will be signed into the application and taken to a landing page that prompts you to begin adding projects:

You can now make some simple changes to get GitLab set up the way you'd like.

Adjusting your Profile Settings

One of the first things you should do after a fresh installation is get your profile into better shape. GitLab selects some reasonable defaults, but these are not usually appropriate once you start using the software.

To make the necessary modifications, click on the user icon in the upper-right hand corner of the interface. In the drop down menu that appears, select Settings:

You will be taken to the Profile section of your settings:

Adjust the Name and Email address from "Administrator" and "admin@example.com" to something more accurate. The name you select will be displayed to other users, while the email will be used for default avatar detection, notifications, Git actions through the interface, etc.

Click on the Update Profile settings button at the bottom when you are done:

A confirmation email will be sent to the address you provided. Follow the instructions in the email to confirm your account so that you can begin using it with GitLab.

Changing Your Account Name

Next, click on the Account item in the left-hand menu bar:

Here, you can find your private API token or configure two-factor authentication. However, the functionality we are interested in at the moment is the Change username section.

By default, the first administrative account is given the name root. Since this is a known account name, it is more secure to change this to a different name. You will still have administrative privileges; the only thing that will change is the name. Replace root with your preferred username:

Click on the Update username button to make the change:

Next time you log in to the GitLab, remember to use your new username.

Adding an SSH Key to your Account

In most cases, you will want to use SSH keys with Git to interact with your GitLab projects. To do this, you need to add your SSH public key to your GitLab account.

If you already have an SSH key pair created on your local computer, you can usually view the public key by typing:

cat ~/.ssh/id_rsa.pub

You should see a large chunk of text, like this:

Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMuyMtMl6aWwqBCvQx7YXvZd7bCFVDsyln3yh5/8Pu23LW88VXfJgsBvhZZ9W0rPBGYyzE/TDzwwITvVQcKrwQrvQlYxTVbqZQDlmsC41HnwDfGFXg+QouZemQ2YgMeHfBzy+w26/gg480nC2PPNd0OG79+e7gFVrTL79JA/MyePBugvYqOAbl30h7M1a7EHP3IV5DQUQg4YUq49v4d3AvM0aia4EUowJs0P/j83nsZt8yiE2JEYR03kDgT/qziPK7LnVFqpFDSPC3MR3b8B354E9Af4C/JHgvglv2tsxOyvKupyZonbyr68CqSorO2rAwY/jWFEiArIaVuDiR9YM5 sammy@mydesktop

Copy this text and head back to the Settings page in GitLab's web interface.

If, instead, you get a message that looks like this, you do not yet have an SSH key pair configured on your machine:

Outputcat: /home/sammy/.ssh/id_rsa.pub: No such file or directory

If this is the case, you can create an SSH key pair by typing:

ssh-keygen

Accept the defaults and optionally provide a password to secure the key locally:

OutputGenerating public/private rsa key pair.
Enter file in which to save the key (/home/sammy/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sammy/.ssh/id_rsa.
Your public key has been saved in /home/sammy/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:I8v5/M5xOicZRZq/XRcSBNxTQV2BZszjlWaIHi5chc0 sammy@gitlab.docsthat.work
The key's randomart image is:
+---[RSA 2048]----+
|          ..%o==B|
|           *.E =.|
|        . ++= B  |
|         ooo.o . |
|      . S .o  . .|
|     . + .. .   o|
|      +   .o.o ..|
|       o .++o .  |
|        oo=+     |
+----[SHA256]-----+

Once you have this, you can display your public key as above by typing:

cat ~/.ssh/id_rsa.pub

Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMuyMtMl6aWwqBCvQx7YXvZd7bCFVDsyln3yh5/8Pu23LW88VXfJgsBvhZZ9W0rPBGYyzE/TDzwwITvVQcKrwQrvQlYxTVbqZQDlmsC41HnwDfGFXg+QouZemQ2YgMeHfBzy+w26/gg480nC2PPNd0OG79+e7gFVrTL79JA/MyePBugvYqOAbl30h7M1a7EHP3IV5DQUQg4YUq49v4d3AvM0aia4EUowJs0P/j83nsZt8yiE2JEYR03kDgT/qziPK7LnVFqpFDSPC3MR3b8B354E9Af4C/JHgvglv2tsxOyvKupyZonbyr68CqSorO2rAwY/jWFEiArIaVuDiR9YM5 sammy@mydesktop

Copy the block of text that's displayed and head back to your Settings in GitLab's web interface.

Click on the SSH Keys item in the left-hand menu:

In the provided space paste the public key you copied from your local machine. Give it a descriptive title, and click the Add key button:

You should now be able to manage your GitLab projects and repositories from your local machine without having to provide your GitLab account credentials.

Step 6 — Restricting or Disabling Public Sign-ups (Optional)

You may have noticed that it is possible for anyone to sign up for an account when you visit your GitLab instance's landing page. This may be what you want if you are looking to host public project. However, many times, more restrictive settings are desirable.

To begin, make your way to the administrative area by clicking on the wrench icon in the main menu bar at the top of the page:

On the page that follows, you can see an overview of your GitLab instance as a whole. To adjust the settings, click on the Settings item at the bottom of the left-hand menu:

You will be taken to the global settings for your GitLab instance. Here, you can adjust a number of settings that affect whether new users can sign up and their level of access.

Disabling Sign-ups

If you wish to disable sign-ups completely (you can still manually create accounts for new users), scroll down to the Sign-up Restrictions section.

Deselect the Sign-up enabled check box:

Scroll down to the bottom and click on the Save changes button:

The sign-up section should now be removed from the GitLab landing page.

Restricting Sign-ups By Domain

If you are using GitLab as part of an organization that provides email addresses associated with a domain, you can restrict sign-ups by domain instead of completely disabling them.

In the Sign-up Restrictions section, select the Send confirmation email on sign-up box, which will allow users to log in only after they've confirmed their email.

Next, add your domain or domains to the Whitelisted domains for sign-ups box, one domain per line. You can use the asterisk "*" to specify wildcard domains:

Scroll down to the bottom and click on the Save changes button:

The sign-up section should now be removed from the GitLab landing page.

Restricting Project Creation

By default, new users can create up to 10 projects. If you wish to allow new users from the outside for visibility and participation, but want to restrict their access to creating new projects, you can do so in the Account and Limit Settings section.

Inside, you can change the Default projects limit to 0 to completely disable new users from creating projects:

New users can still be added to projects manually and will have access to internal or public projects created by other users.

Scroll down to the bottom and click on the Save changes button:

New users will now be able to create accounts, but unable to create projects.

Renewing Let's Encrypt Certificates

By default, GitLab has a scheduled task set up to renew Let's Encrypt certificates after midnight every fourth day, with the exact minute based on your external_url. You can modify these settings in the /etc/gitlab/gitlab.rb file. For example, if you wanted to renew every 7th day at 12:30, you could configure this as follows:

/etc/gitlab/gitlab.rb

letsencrypt['auto_renew_hour'] = "12"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"

You can also disable auto-renewal by adding an additional setting to /etc/gitlab/gitlab.rb:

/etc/gitlab/gitlab.rb

letsencrypt['auto_renew'] = false

With auto-renewals in place, you will not need to worry about service interruptions.

Conclusion

You should now have a working GitLab instance hosted on your own server. You can begin to import or create new projects and configure the appropriate level of access for your team. GitLab is regularly adding features and making updates to their platform, so be sure to check out the project's home page to stay up-to-date on any improvements or important notices.

How To Set Up vsftpd for a User's Directory on Debian 9

2018-09-07T13:48:13Z

Introduction

FTP, short for File Transfer Protocol, is a network protocol that was once widely used for moving files between a client and server. It has since been replaced by faster, more secure, and more convenient ways of delivering files. Many casual internet users expect to download directly from their web browser with https, and command-line users are more likely to use secure protocols such as the scp or SFTP.

FTP is still used to support legacy applications and workflows with very specific needs. If you have a choice of what protocol to use, consider exploring the more modern options. When you do need FTP, however, vsftpd is an excellent choice. Optimized for security, performance, and stability, vsftpd offers strong protection against many security problems found in other FTP servers and is the default for many Linux distributions.

In this tutorial, you'll configure vsftpd to allow a user to upload files to his or her home directory using FTP with login credentials secured by SSL/TLS.

Prerequisites

To follow along with this tutorial you will need:

A Debian 9 server, and a non-root user with sudo privileges. You can learn more about how to create a user with these privileges in our Initial Server Setup with Debian 9 guide.

Step 1 — Installing vsftpd

Let's start by updating our package list and installing the vsftpd daemon:

sudo apt update
sudo apt install vsftpd

When the installation is complete, let's copy the configuration file so we can start with a blank configuration, and save the original as a backup:

sudo cp /etc/vsftpd.conf /etc/vsftpd.conf.orig

With a backup of the configuration in place, we're ready to configure the firewall.

Step 2 — Opening the Firewall

Let's check the firewall status to see if it’s enabled. If it is, we’ll ensure that FTP traffic is permitted so firewall rules don't block our tests. This guide assumes that you have UFW installed, following Step 4 in the initial server setup guide.

Check the firewall status:

sudo ufw status

In this case, only SSH is allowed through:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

You may have other rules in place or no firewall rules at all. Since only SSH traffic is permitted in this case, we’ll need to add rules for FTP traffic.

Let's open ports 20 and 21 for FTP, port 990 for when we enable TLS, and ports 40000-50000 for the range of passive ports we plan to set in the configuration file:

sudo ufw allow 20/tcp
sudo ufw allow 21/tcp
sudo ufw allow 990/tcp
sudo ufw allow 40000:50000/tcp

Check the firewall status:

sudo ufw status

Your firewall rules should now look like this:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
990/tcp                    ALLOW       Anywhere
20/tcp                     ALLOW       Anywhere
21/tcp                     ALLOW       Anywhere
40000:50000/tcp            ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
20/tcp (v6)                ALLOW       Anywhere (v6)
21/tcp (v6)                ALLOW       Anywhere (v6)
990/tcp (v6)               ALLOW       Anywhere (v6)
40000:50000/tcp (v6)       ALLOW       Anywhere (v6)

With vsftpd installed and the necessary ports open, let's move on to creating a dedicated FTP user.

Step 3 — Preparing the User Directory

We will create a dedicated FTP user, but you may already have a user in need of FTP access. We'll take care to preserve an existing user’s access to their data in the instructions that follow. Even so, we recommend that you start with a new user until you've configured and tested your setup.

First, add a test user:

sudo adduser sammy

Assign a password when prompted. Feel free to press ENTER through the other prompts.

FTP is generally more secure when users are restricted to a specific directory. vsftpd accomplishes this with chroot jails. When chroot is enabled for local users, they are restricted to their home directory by default. However, because of the way vsftpd secures the directory, it must not be writable by the user. This is fine for a new user who should only connect via FTP, but an existing user may need to write to their home folder if they also have shell access.

In this example, rather than removing write privileges from the home directory, let's create an ftp directory to serve as the chroot and a writable files directory to hold the actual files.

Create the ftp folder:

sudo mkdir /home/sammy/ftp

Set its ownership:

sudo chown nobody:nogroup /home/sammy/ftp

Remove write permissions:

sudo chmod a-w /home/sammy/ftp

Verify the permissions:

sudo ls -la /home/sammy/ftp

Outputtotal 8
4 dr-xr-xr-x  2 nobody nogroup 4096 Aug 24 21:29 .
4 drwxr-xr-x  3 sammy  sammy   4096 Aug 24 21:29 ..

Next, let's create the directory for file uploads and assign ownership to the user:

sudo mkdir /home/sammy/ftp/files
sudo chown sammy:sammy /home/sammy/ftp/files

A permissions check on the ftp directory should return the following:

sudo ls -la /home/sammy/ftp

Outputtotal 12
dr-xr-xr-x 3 nobody nogroup 4096 Aug 26 14:01 .
drwxr-xr-x 3 sammy  sammy   4096 Aug 26 13:59 ..
drwxr-xr-x 2 sammy  sammy   4096 Aug 26 14:01 files

Finally, let's add a test.txt file to use when we test:

echo "vsftpd test file" | sudo tee /home/sammy/ftp/files/test.txt

Now that we've secured the ftp directory and allowed the user access to the files directory, let's modify our configuration.

Step 4 — Configuring FTP Access

We're planning to allow a single user with a local shell account to connect with FTP. The two key settings for this are already set in vsftpd.conf. Start by opening the config file to verify that the settings in your configuration match those below:

sudo nano /etc/vsftpd.conf

/etc/vsftpd.conf

. . .
# Allow anonymous FTP? (Disabled by default).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
local_enable=YES
. . .

Next, let's enable the user to upload files by uncommenting the write_enable setting:

/etc/vsftpd.conf

. . .
write_enable=YES
. . .

We’ll also uncomment the chroot to prevent the FTP-connected user from accessing any files or commands outside the directory tree:

/etc/vsftpd.conf

. . .
chroot_local_user=YES
. . .

Let's also add a user_sub_token to insert the username in our local_root directory path so our configuration will work for this user and any additional future users. Add these settings anywhere in the file:

/etc/vsftpd.conf

. . .
user_sub_token=$USER
local_root=/home/$USER/ftp

Let's also limit the range of ports that can be used for passive FTP to make sure enough connections are available:

/etc/vsftpd.conf

. . .
pasv_min_port=40000
pasv_max_port=50000

Note: In Step 2, we opened the ports that we set here for the passive port range. If you change the values, be sure to update your firewall settings.

To allow FTP access on a case-by-case basis, let's set the configuration so that users only have access when they are explicitly added to a list, rather than by default:

/etc/vsftpd.conf

. . .
userlist_enable=YES
userlist_file=/etc/vsftpd.userlist
userlist_deny=NO

userlist_deny toggles the logic: When it is set to YES, users on the list are denied FTP access. When it is set to NO, only users on the list are allowed access.

When you're done making the changes, save the file and exit the editor.

Finally, let's add our user to /etc/vsftpd.userlist. Use the -a flag to append to the file:

echo "sammy" | sudo tee -a /etc/vsftpd.userlist

Check that it was added as you expected:

cat /etc/vsftpd.userlist

Outputsammy

Restart the daemon to load the configuration changes:

sudo systemctl restart vsftpd

With the configuration in place, let's move on to testing FTP access.

Step 5 — Testing FTP Access

We've configured the server to allow only the user sammy to connect via FTP. Let's make sure that this works as expected.

Anonymous users should fail to connect: We've disabled anonymous access. Let's test that by trying to connect anonymously. If our configuration is set up properly, anonymous users should be denied permission. Open another terminal and run the following command. Be sure to replace 203.0.113.0 with your server's public IP address:

ftp -p 203.0.113.0

OutputConnected to 203.0.113.0.
220 (vsFTPd 3.0.3)
Name (203.0.113.0:default): anonymous
530 Permission denied.
ftp: Login failed.
ftp>

Close the connection:

bye

Users other than sammy should fail to connect: Next, let's try connecting as our sudo user. They should also be denied access, and it should happen before they're allowed to enter their password:

ftp -p 203.0.113.0

OutputConnected to 203.0.113.0.
220 (vsFTPd 3.0.3)
Name (203.0.113.0:default): your_sudo_user
530 Permission denied.
ftp: Login failed.
ftp>

Close the connection:

bye

The user sammy should be able to connect, read, and write files: Let's make sure that our designated user can connect:

ftp -p 203.0.113.0

OutputConnected to 203.0.113.0.
220 (vsFTPd 3.0.3)
Name (203.0.113.0:default): sammy
331 Please specify the password.
Password: your_user's_password
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

Let's change into the files directory and use the get command to transfer the test file we created earlier to our local machine:

cd files
get test.txt

Output229 Entering Extended Passive Mode (|||47398|)
150 Opening BINARY mode data connection for test.txt (17 bytes).
100% |**********************************|    17      146.91 KiB/s    00:00 ETA
226 Transfer complete.
17 bytes received in 00:00 (0.17 KiB/s)
ftp>

Next, let's upload the file with a new name to test write permissions:

put test.txt upload.txt

Output229 Entering Extended Passive Mode (|||46598|)
150 Ok to send data.
100% |**********************************|    17        8.93 KiB/s    00:00 ETA
226 Transfer complete.
17 bytes sent in 00:00 (0.08 KiB/s)

Close the connection:

bye

Now that we've tested our configuration, let's take steps to further secure our server.

Step 6 — Securing Transactions

Since FTP does not encrypt any data in transit, including user credentials, we'll enable TLS/SSL to provide that encryption. The first step is to create the SSL certificates for use with vsftpd.

Let's use openssl to create a new certificate and use the -days flag to make it valid for one year. In the same command, we'll add a private 2048-bit RSA key. By setting both the -keyout and -out flags to the same value, the private key and the certificate will be located in the same file:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/vsftpd.pem -out /etc/ssl/private/vsftpd.pem

You'll be prompted to provide address information for your certificate. Substitute your own information for the highlighted values below:

OutputGenerating a 2048 bit RSA private key
............................................................................+++
...........+++
writing new private key to '/etc/ssl/private/vsftpd.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:DigitalOcean
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []: your_server_ip
Email Address []:

For more detailed information about the certificate flags, see OpenSSL Essentials: Working with SSL Certificates, Private Keys and CSRs

Once you've created the certificates, open the vsftpd configuration file again:

sudo nano /etc/vsftpd.conf

Toward the bottom of the file, you will see two lines that begin with rsa_. Comment them out so they look like this:

/etc/vsftpd.conf

. . .
# rsa_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
# rsa_private_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
. . .

Below them, add the following lines that point to the certificate and private key we just created:

/etc/vsftpd.conf

. . .
rsa_cert_file=/etc/ssl/private/vsftpd.pem
rsa_private_key_file=/etc/ssl/private/vsftpd.pem
. . .

After that, we will force the use of SSL, which will prevent clients that can't deal with TLS from connecting. This is necessary to ensure that all traffic is encrypted, but it may force your FTP user to change clients. Change ssl_enable to YES:

/etc/vsftpd.conf

. . .
ssl_enable=YES
. . .

After that, add the following lines to explicitly deny anonymous connections over SSL and to require SSL for both data transfer and logins:

/etc/vsftpd.conf

. . .
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
. . .

After this, configure the server to use TLS, the preferred successor to SSL, by adding the following lines:

/etc/vsftpd.conf

. . .
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
. . .

Finally, we will add two more options. First, we will not require SSL reuse because it can break many FTP clients. We will require "high" encryption cipher suites, which currently means key lengths equal to or greater than 128 bits:

/etc/vsftpd.conf

. . .
require_ssl_reuse=NO
ssl_ciphers=HIGH
. . .

The finished file section should look like this:

/etc/vsftpd.conf

# This option specifies the location of the RSA certificate to use for SSL
# encrypted connections.
#rsa_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
#rsa_private_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
rsa_cert_file=/etc/ssl/private/vsftpd.pem
rsa_private_key_file=/etc/ssl/private/vsftpd.pem
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH

When you're done, save and close the file.

Restart the server for the changes to take effect:

sudo systemctl restart vsftpd

At this point, we will no longer be able to connect with an insecure command-line client. If we tried, we'd see something like:

Output
ftp -p 203.0.113.0
Connected to 203.0.113.0.
220 (vsFTPd 3.0.3)
Name (203.0.113.0:default): sammy
530 Non-anonymous sessions must use encryption.
ftp: Login failed.
ftp>

Next, let's verify that we can connect using a client that supports TLS.

Step 7 — Testing TLS with FileZilla

Most modern FTP clients can be configured to use TLS encryption. We will demonstrate how to connect with FileZilla because of its cross-platform support. Consult the documentation for other clients.

When you first open FileZilla, find the Site Manager icon just above the word Host, the left-most icon on the top row. Click it:

A new window will open. Click the New Site button in the bottom right corner:

Under My Sites a new icon with the words New Site will appear. You can name it now or return later and use the Rename button.

Fill out the Host field with the name or IP address. Under the Encryption drop down menu, select Require explicit FTP over TLS.

For Logon Type, select Ask for password. Fill in your FTP user in the User field:

Click Connect at the bottom of the interface. You will be asked for the user's password:

Click OK to connect. You should now be connected with your server with TLS/SSL encryption.

Upon success, you will be presented with a server certificate that looks like this:

When you’ve accepted the certificate, double-click the files folder and drag upload.txt to the left to confirm that you’re able to download files:

When you’ve done that, right-click on the local copy, rename it to upload-tls.txt and drag it back to the server to confirm that you can upload files:

You’ve now confirmed that you can securely and successfully transfer files with SSL/TLS enabled.

Step 8 — Disabling Shell Access (Optional)

If you're unable to use TLS because of client requirements, you can gain some security by disabling the FTP user's ability to log in any other way. One relatively straightforward way to prevent it is by creating a custom shell. This will not provide any encryption, but it will limit the access of a compromised account to files accessible by FTP.

First, open a file called ftponly in the bin directory:

sudo nano /bin/ftponly

Add a message telling the user why they are unable to log in:

/bin/ftponly

#!/bin/sh
echo "This account is limited to FTP access only."

Save the file and exit your editor.

Change the permissions to make the file executable:

sudo chmod a+x /bin/ftponly

Open the list of valid shells:

sudo nano /etc/shells

At the bottom add:

/etc/shells

. . .
/bin/ftponly

Update the user's shell with the following command:

sudo usermod sammy -s /bin/ftponly

Now try logging into your server as sammy:

ssh sammy@your_server_ip

You should see something like:

OutputThis account is limited to FTP access only.
Connection to 203.0.113.0 closed.

This confirms that the user can no longer ssh to the server and is limited to FTP access only.

Conclusion

In this tutorial we covered setting up FTP for users with a local account. If you need to use an external authentication source, you might want to look into vsftpd's support of virtual users. This offers a rich set of options through the use of PAM, the Pluggable Authentication Modules, and is a good choice if you manage users in another system such as LDAP or Kerberos.

How To Install Java with Apt on Debian 9

2018-09-07T04:44:34Z

Introduction

Java and the JVM (Java's virtual machine) are required for many kinds of software, including Tomcat, Jetty, Glassfish, Cassandra and Jenkins.

In this guide, you will install various versions of the Java Runtime Environment (JRE) and the Java Developer Kit (JDK) using apt . You'll install OpenJDK as well as official packages from Oracle. You'll then select the version you wish to use for your projects. When you're finished, you'll be able to use the JDK to develop software or use the Java Runtime to run software.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server set up by following the the Debian 9 initial server setup guide tutorial, including a non-root user with sudo access and a firewall.

Installing the Default JRE/JDK

The easiest option for installing Java is to use the version packaged with Debian. By default, Debian 9 includes Open JDK, which is an open-source variant of the JRE and JDK.

This package will install OpenJDK version 1.8, which is compatible with Java 8. Java 8 is the current Long Term Support version and is still widely supported, though public maintenance ends in January 2019.

To install this version, first update the package index:

sudo apt update

Next, check if Java is already installed:

java -version

If Java is not currently installed, you'll see the following output:

Output-bash: java: command not found

Execute the following command to install OpenJDK:

sudo apt install default-jre

This command will install the Java Runtime Environment (JRE). This will allow you to run almost all Java software.

Verify the installation with:

java -version

You'll see the following output:

Outputopenjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

You may need the Java Development Kit (JDK) in addition to the JRE in order to compile and run some specific Java-based software. To install the JDK, execute the following command, which will also install the JRE:

sudo apt install default-jdk

Verify that the JDK is installed by checking the version of javac, the Java compiler:

javac -version

You'll see the following output:

Outputjavac 1.8.0_181

Next, let's look at how to install Oracle's official JDK and JRE.

Installing the Oracle JDK

If you want to install the Oracle JDK, which is the official version distributed by Oracle, you'll need to add a new package repository for the version you'd like to use.

First, install the software-properties-common package which adds the apt-get-repository command which you'll use to add additional repositories to your sources list.

Install software-properties-common with:

sudo apt install software-properties-common

With this installed, you can install Oracle's Java.

Installing Oracle Java 8

To install Java 8, which is the current long-term support version, first add its package repository:

sudo add-apt-repository ppa:webupd8team/java

When you add the repository, you'll see a message like this:

output Oracle Java (JDK) Installer (automatically downloads and installs Oracle JDK8). There are no actual Java files in this PPA.

Important -> Why Oracle Java 7 And 6 Installers No Longer Work: http://www.webupd8.org/2017/06/why-oracle-java-7-and-6-installers-no.html

Update: Oracle Java 9 has reached end of life: http://www.oracle.com/technetwork/java/javase/downloads/jdk9-downloads-3848520.html

The PPA supports Ubuntu 18.04, 17.10, 16.04, 14.04 and 12.04.

More info (and Ubuntu installation instructions):
- for Oracle Java 8: http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

Debian installation instructions:
- Oracle Java 8: http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html

For Oracle Java 10, see a different PPA: https://www.linuxuprising.com/2018/04/install-oracle-java-10-in-ubuntu-or.html
 More info: https://launchpad.net/~webupd8team/+archive/ubuntu/java
Press [ENTER] to continue or ctrl-c to cancel adding it

Press ENTER to continue. It will attempt to import some GPG signing keys, but it won't be able to find any valid ones:

Outputgpg: keybox '/tmp/tmpgt9wdvth/pubring.gpg' created
gpg: /tmp/tmpgt9wdvth/trustdb.gpg: trustdb created
gpg: key C2518248EEA14886: public key "Launchpad VLC" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
gpg: no valid OpenPGP data found.

Execute the following command to add the GPG key for the repository source manually:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C2518248EEA14886

Then update your package list:

sudo apt update

Once the package list updates, install Java 8:

sudo apt install oracle-java8-installer

Your system will download the JDK from Oracle and ask you to accept the license agreement. Accept the agreement and the JDK will install.

Installing Oracle Java 10

To install Oracle Java 10, first add its repository:

sudo add-apt-repository ppa:linuxuprising/java

You'll see this message:

Output Oracle Java 10 installer

Java binaries are not hosted in this PPA due to licensing. The packages in this PPA download and install Oracle Java 10 (JDK 10), so a working Internet connection is required.

The packages in this PPA are based on the WebUpd8 Oracle Java PPA packages: https://launchpad.net/~webupd8team/+archive/ubuntu/java

Created for users of https://www.linuxuprising.com/

Issues or suggestions? Leave a comment here: https://www.linuxuprising.com/2018/04/install-oracle-java-10-in-ubuntu-or.html
 More info: https://launchpad.net/~linuxuprising/+archive/ubuntu/java
Press [ENTER] to continue or ctrl-c to cancel adding it

Press ENTER to continue the installation. Like with Java 8, you'll see a message about invalid signing keys:

Outputgpg: keybox '/tmp/tmpvuqsh9ui/pubring.gpg' created
gpg: /tmp/tmpvuqsh9ui/trustdb.gpg: trustdb created
gpg: key EA8CACC073C3DB2A: public key "Launchpad PPA for Linux Uprising" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: no valid OpenPGP data found.

Execute this command to import the necessary key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EA8CACC073C3DB2A

Then update your package list:

sudo apt update

Once the package list updates, install Java 10:

sudo apt install oracle-java10-installer

Your system will download the JDK from Oracle and ask you to accept the license agreement. Accept the agreement and the JDK will install.

Now let's look at how to select which version of Java you want to use.

Managing Java

You can have multiple Java installations on one server. You can configure which version is the default for use on the command line by using the update-alternatives command.

sudo update-alternatives --config java

This is what the output would look like if you've installed all versions of Java in this tutorial:

OutputThere are 3 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-10-oracle/bin/java             1091      auto mode
* 1            /usr/lib/jvm/java-10-oracle/bin/java             1091      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode
  3            /usr/lib/jvm/java-8-oracle/jre/bin/java          1081      manual mode

Press <enter> to keep the current choice[*], or type selection number:

Choose the number associated with the Java version to use it as the default, or press ENTER to leave the current settings in place.

You can do this for other Java commands, such as the compiler (javac):

sudo update-alternatives --config javac

Other commands for which this command can be run include, but are not limited to: keytool, javadoc and jarsigner.

Let's set the JAVA_HOME environment variable next.

Setting the `JAVA_HOME` Environment Variable

Many programs written using Java use the JAVA_HOME environment variable to determine the Java installation location.

To set this environment variable, first determine where Java is installed. Use the update-alternatives command again:

sudo update-alternatives --config java

This command shows each installation of Java along with its installation path:

Output  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-10-oracle/bin/java             1091      auto mode
* 1            /usr/lib/jvm/java-10-oracle/bin/java             1091      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode
  3            /usr/lib/jvm/java-8-oracle/jre/bin/java          1081      manual mode

In this case the installation paths are as follows:

Oracle Java 10 is located at /usr/lib/jvm/java-10-oracle/jre/bin/java.
Oracle Java 8 is located at /usr/lib/jvm/java-8-oracle/jre/bin/java.
OpenJDK 8 is located at /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java.

These paths show the path to the java executable.

Copy the path for your preferred installation, excluding the trailing bin/java component. Then open /etc/environment using nano or your favorite text editor:

sudo nano /etc/environment

At the end of this file, add the following line, making sure to replace the highlighted path with your own copied path:

/etc/environment

JAVA_HOME="/usr/lib/jvm/java-8-oracle/jre"

Modifying this file will set the JAVA_HOME path for all users on your system.

Save the file and exit the editor.

Now reload this file to apply the changes to your current session:

source /etc/environment

Verify that the environment variable is set:

echo $JAVA_HOME

You'll see the path you just set:

Output
/usr/lib/jvm/java-8-oracle/jre

Other users will need to execute the command source /etc/environment or log out and log back in to apply this setting.

Conclusion

In this tutorial you installed multiple versions of Java and learned how to manage them. You can now install software which runs on Java, such as Tomcat, Jetty, Glassfish, Cassandra or Jenkins.

How To Install Docker Compose on Debian 9

2018-09-06T22:13:28Z

Introduction

Docker is a great tool for automating the deployment of Linux applications inside software containers, but to take full advantage of its potential each component of an application should run in its own individual container. For complex applications with a lot of components, orchestrating all the containers to start up, communicate, and shut down together can quickly become unwieldy.

The Docker community came up with a popular solution called Fig, which allowed you to use a single YAML file to orchestrate all of your Docker containers and configurations. This became so popular that the Docker team decided to make Docker Compose based on the Fig source, which is now deprecated. Docker Compose makes it easier for users to orchestrate the processes of Docker containers, including starting up, shutting down, and setting up intra-container linking and volumes.

In this tutorial, we'll show you how to install the latest version of Docker Compose to help you manage multi-container applications on a Debian 9 server.

Prerequisites

To follow this article, you will need:

A Debian 9 server and a non-root user with sudo privileges. This initial server setup with Debian 9 tutorial explains how to set this up.
Docker installed with the instructions from Step 1 and Step 2 of How To Install and Use Docker on Debian 9

Note: Even though the Prerequisites give instructions for installing Docker on Debian 9, the docker commands in this article should work on other operating systems as long as Docker is installed.

Step 1 — Installing Docker Compose

Although we can install Docker Compose from the official Debian repositories, it is several minor versions behind the latest release, so we'll install it from Docker's GitHub repository. The command below is slightly different than the one you'll find on the Releases page. By using the -o flag to specify the output file first rather than redirecting the output, this syntax avoids running into a permission denied error caused when using sudo.

We'll check the current release and, if necessary, update it in the command below:

sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

Next we'll set the permissions:

sudo chmod +x /usr/local/bin/docker-compose

Then we'll verify that the installation was successful by checking the version:

docker-compose --version

This will print out the version we installed:

Output
docker-compose version 1.22.0, build f46880fe

Now that we have Docker Compose installed, we're ready to run a "Hello World" example.

Step 2 — Running a Container with Docker Compose

The public Docker registry, Docker Hub, includes a Hello World image for demonstration and testing. It illustrates the minimal configuration required to run a container using Docker Compose: a YAML file that calls a single image. We'll create this minimal configuration to run our hello-world container.

First, we'll create a directory for the YAML file and move into it:

mkdir hello-world
cd hello-world

Then, we'll create the YAML file:

nano docker-compose.yml

Put the following contents into the file, save the file, and exit the text editor:

docker-compose.yml

my-test:
 image: hello-world

The first line in the YAML file is used as part of the container name. The second line specifies which image to use to create the container. When we run the docker-compose up command, it will look for a local image by the name we specified, hello-world. With this in place, we’ll save and exit the file.

We can look manually at images on our system with the docker images command:

docker images

When there are no local images at all, only the column headings display:

OutputREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

Now, while still in the ~/hello-world directory, we'll execute the following command:

docker-compose up

The first time we run the command, if there's no local image named hello-world, Docker Compose will pull it from the Docker Hub public repository:

OutputPulling my-test (hello-world:)...
latest: Pulling from library/hello-world
9db2ca6ccae0: Pull complete
Digest: sha256:4b8ff392a12ed9ea17784bd3c9a8b1fa3299cac44aca35a85c90c5e3c7afacdc
Status: Downloaded newer image for hello-world:latest
. . .

After pulling the image, docker-compose creates a container, attaches, and runs the hello program, which in turn confirms that the installation appears to be working:

Output. . .
Creating helloworld_my-test_1...
Attaching to helloworld_my-test_1
my-test_1 |
my-test_1 | Hello from Docker.
my-test_1 | This message shows that your installation appears to be working correctly.
my-test_1 |
. . .

Then it prints an explanation of what it did:

Output To generate this message, Docker took the following steps:
my-test_1  |  1. The Docker client contacted the Docker daemon.
my-test_1  |  2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
my-test_1  |     (amd64)
my-test_1  |  3. The Docker daemon created a new container from that image which runs the
my-test_1  |     executable that produces the output you are currently reading.
my-test_1  |  4. The Docker daemon streamed that output to the Docker client, which sent it
my-test_1  |     to your terminal.

Docker containers only run as long as the command is active, so once hello finished running, the container stopped. Consequently, when we look at active processes, the column headers will appear, but the hello-world container won't be listed because it's not running:

docker ps

OutputCONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES

We can see the container information, which we'll need in the next step, by using the -a flag. This shows all containers, not just active ones:

docker ps -a

OutputCONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
06069fd5ca23        hello-world         "/hello"            35 minutes ago      Exited (0) 35 minutes ago                       hello-world_my-test_1

This displays the information we'll need to remove the container when we're done with it.

Step 3 — Removing the Image (Optional)

To avoid using unnecessary disk space, we'll remove the local image. To do so, we'll need to delete all the containers that reference the image using the docker rm command, followed by either the CONTAINER ID or the NAME. Below, we're using the CONTAINER ID from the docker ps -a command we just ran. Be sure to substitute the ID of your container:

docker rm 06069fd5ca23

Once all containers that reference the image have been removed, we can remove the image:

docker rmi hello-world

Conclusion

We've now installed Docker Compose, tested our installation by running a Hello World example, and removed the test image and container.

While the Hello World example confirmed our installation, the simple configuration does not show one of the main benefits of Docker Compose — being able to bring a group of Docker containers up and down all at the same time. To see the power of Docker Compose in action, you might like to check out this practical example, How To Configure a Continuous Integration Testing Environment with Docker and Docker Compose on Ubuntu 16.04.

How To Set Up a Firewall with UFW on Debian 9

2018-09-06T20:28:19Z

Introduction

UFW, or Uncomplicated Firewall, is an interface to iptables that is geared towards simplifying the process of configuring a firewall. While iptables is a solid and flexible tool, it can be difficult for beginners to learn how to use it to properly configure a firewall. If you're looking to get started securing your network, and you're not sure which tool to use, UFW may be the right choice for you.

This tutorial will show you how to set up a firewall with UFW on Debian 9.

Prerequisites

To follow this tutorial, you will need One Debian 9 server with a sudo non-root user, which you can set up by following Steps 1–3 in the Initial Server Setup with Debian 9 tutorial.

Step 1 – Installing UFW

Debian does not install UFW by default. If you followed through the entire Initial Server Setup tutorial you will have already installed and enabled UFW. If not, install it now using apt:

sudo apt install ufw

We will set up UFW and enable it in the following steps.

Step 2 — Using IPv6 with UFW (Optional)

This tutorial is written with IPv4 in mind, but will work for IPv6 as well as long as you enable it. If your Debian server has IPv6 enabled, ensure that UFW is configured to support IPv6 so that it will manage firewall rules for IPv6 in addition to IPv4. To do this, open the UFW configuration with nano or your favorite editor.

sudo nano /etc/default/ufw

Then make sure the value of IPV6 is yes. It should look like this:

/etc/default/ufw excerpt

IPV6=yes

Save and close the file. Now, when UFW is enabled, it will be configured to write both IPv4 and IPv6 firewall rules. However, before enabling UFW, we will want to ensure that your firewall is configured to allow you to connect via SSH. Let's start with setting the default policies.

Step 3 — Setting Up Default Policies

If you're just getting started with your firewall, the first rules to define are your default policies. These rules control how to handle traffic that does not explicitly match any other rules. By default, UFW is set to deny all incoming connections and allow all outgoing connections. This means anyone trying to reach your server would not be able to connect, while any application within the server would be able to reach the outside world.

Let's set your UFW rules back to the defaults so we can be sure that you'll be able to follow along with this tutorial. To set the defaults used by UFW, use these commands:

sudo ufw default deny incoming
sudo ufw default allow outgoing

These commands set the defaults to deny incoming and allow outgoing connections. These firewall defaults alone might suffice for a personal computer, but servers typically need to respond to incoming requests from outside users. We'll look into that next.

Step 4 — Allowing SSH Connections

If we enabled our UFW firewall now, it would deny all incoming connections. This means that we will need to create rules that explicitly allow legitimate incoming connections — SSH or HTTP connections, for example — if we want our server to respond to those types of requests. If you're using a cloud server, you will probably want to allow incoming SSH connections so you can connect to and manage your server.

To configure your server to allow incoming SSH connections, you can use this command:

sudo ufw allow ssh

This will create firewall rules that will allow all connections on port 22, which is the port that the SSH daemon listens on by default. UFW knows what port allow ssh means because it's listed as a service in the /etc/services file.

However, we can actually write the equivalent rule by specifying the port instead of the service name. For example, this command works the same as the one above:

sudo ufw allow 22

If you configured your SSH daemon to use a different port, you will have to specify the appropriate port. For example, if your SSH server is listening on port 2222, you can use this command to allow connections on that port:

sudo ufw allow 2222

Now that your firewall is configured to allow incoming SSH connections, we can enable it.

Step 5 — Enabling UFW

To enable UFW, use this command:

sudo ufw enable

You will receive a warning that says the command may disrupt existing SSH connections. We already set up a firewall rule that allows SSH connections, so it should be fine to continue. Respond to the prompt with y and hit ENTER.

The firewall is now active. Run the sudo ufw status verbose command to see the rules that are set. The rest of this tutorial covers how to use UFW in more detail, like allowing or denying different kinds of connections.

Step 6 — Allowing Other Connections

At this point, you should allow all of the other connections that your server needs to respond to. The connections that you should allow depends on your specific needs. Luckily, you already know how to write rules that allow connections based on a service name or port; we already did this for SSH on port 22. You can also do this for:

HTTP on port 80, which is what unencrypted web servers use, using sudo ufw allow http or sudo ufw allow 80
HTTPS on port 443, which is what encrypted web servers use, using sudo ufw allow https or sudo ufw allow 443

There are several others ways to allow other connections, aside from specifying a port or known service.

Specific Port Ranges

You can specify port ranges with UFW. Some applications use multiple ports, instead of a single port.

For example, to allow X11 connections, which use ports 6000-6007, use these commands:

sudo ufw allow 6000:6007/tcp
sudo ufw allow 6000:6007/udp

When specifying port ranges with UFW, you must specify the protocol (tcp or udp) that the rules should apply to. We haven't mentioned this before because not specifying the protocol automatically allows both protocols, which is OK in most cases.

Specific IP Addresses

When working with UFW, you can also specify IP addresses. For example, if you want to allow connections from a specific IP address, such as a work or home IP address of 203.0.113.4, you need to specify from, then the IP address:

sudo ufw allow from 203.0.113.4

You can also specify a specific port that the IP address is allowed to connect to by adding to any port followed by the port number. For example, If you want to allow 203.0.113.4 to connect to port 22 (SSH), use this command:

sudo ufw allow from 203.0.113.4 to any port 22

Subnets

If you want to allow a subnet of IP addresses, you can do so using CIDR notation to specify a netmask. For example, if you want to allow all of the IP addresses ranging from 203.0.113.1 to 203.0.113.254 you could use this command:

sudo ufw allow from 203.0.113.0/24

Likewise, you may also specify the destination port that the subnet 203.0.113.0/24 is allowed to connect to. Again, we'll use port 22 (SSH) as an example:

sudo ufw allow from 203.0.113.0/24 to any port 22

Connections to a Specific Network Interface

If you want to create a firewall rule that only applies to a specific network interface, you can do so by specifying "allow in on" followed by the name of the network interface.

You may want to look up your network interfaces before continuing. To do so, use this command:

ip addr

Output Excerpt
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
. . .
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
. . .

The highlighted output indicates the network interface names. They are typically named something like eth0 or enp3s2.

So, if your server has a public network interface called eth0, you could allow HTTP traffic (port 80) to it with this command:

sudo ufw allow in on eth0 to any port 80

Doing so would allow your server to receive HTTP requests from the public internet.

Or, if you want your MySQL database server (port 3306) to listen for connections on the private network interface eth1, for example, you could use this command:

sudo ufw allow in on eth1 to any port 3306

This would allow other servers on your private network to connect to your MySQL database.

Step 7 — Denying Connections

If you haven't changed the default policy for incoming connections, UFW is configured to deny all incoming connections. Generally, this simplifies the process of creating a secure firewall policy by requiring you to create rules that explicitly allow specific ports and IP addresses through.

However, sometimes you will want to deny specific connections based on the source IP address or subnet, perhaps because you know that your server is being attacked from there. Also, if you want to change your default incoming policy to allow (which is not recommended), you would need to create deny rules for any services or IP addresses that you don't want to allow connections for.

To write deny rules, you can use the commands described above, replacing allow with deny.

For example, to deny HTTP connections, you could use this command:

sudo ufw deny http

Or if you want to deny all connections from 203.0.113.4 you could use this command:

sudo ufw deny from 203.0.113.4

Now let's take a look at how to delete rules.

Step 8 — Deleting Rules

Knowing how to delete firewall rules is just as important as knowing how to create them. There are two different ways to specify which rules to delete: by rule number or by the actual rule (similar to how the rules were specified when they were created). We'll start with the delete by rule number method because it is easier.

By Rule Number

If you're using the rule number to delete firewall rules, the first thing you'll want to do is get a list of your firewall rules. The UFW status command has an option to display numbers next to each rule, as demonstrated here:

sudo ufw status numbered

Numbered Output:Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22                         ALLOW IN    15.15.15.0/24
[ 2] 80                         ALLOW IN    Anywhere

If we decide that we want to delete rule 2, the one that allows port 80 (HTTP) connections, we can specify it in a UFW delete command like this:

sudo ufw delete 2

This would show a confirmation prompt then delete rule 2, which allows HTTP connections. Note that if you have IPv6 enabled, you would want to delete the corresponding IPv6 rule as well.

By Actual Rule

The alternative to rule numbers is to specify the actual rule to delete. For example, if you want to remove the allow http rule, you could write it like this:

sudo ufw delete allow http

You could also specify the rule by allow 80, instead of by service name:

sudo ufw delete allow 80

This method will delete both IPv4 and IPv6 rules, if they exist.

Step 9 — Checking UFW Status and Rules

At any time, you can check the status of UFW with this command:

sudo ufw status verbose

If UFW is disabled, which it is by default, you'll see something like this:

OutputStatus: inactive

If UFW is active, which it should be if you followed Step 3, the output will say that it's active and it will list any rules that are set. For example, if the firewall is set to allow SSH (port 22) connections from anywhere, the output might look something like this:

OutputStatus: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere

Use the status command if you want to check how UFW has configured the firewall.

Step 10 — Disabling or Resetting UFW (optional)

If you decide you don't want to use UFW, you can disable it with this command:

sudo ufw disable

Any rules that you created with UFW will no longer be active. You can always run sudo ufw enable if you need to activate it later.

If you already have UFW rules configured but you decide that you want to start over, you can use the reset command:

sudo ufw reset

This will disable UFW and delete any rules that were previously defined. Keep in mind that the default policies won't change to their original settings, if you modified them at any point. This should give you a fresh start with UFW.

Conclusion

Your firewall is now configured to allow (at least) SSH connections. Be sure to allow any other incoming connections that your server, while limiting any unnecessary connections, so your server will be functional and secure.

To learn about more common UFW configurations, check out the UFW Essentials: Common Firewall Rules and Commands tutorial.

How To Set Up Django with Postgres, Nginx, and Gunicorn on Debian 9

2018-09-06T19:43:12Z

Introduction

Django is a powerful web framework that can help you get your Python application or website off the ground. Django includes a simplified development server for testing your code locally, but for anything even slightly production related, a more secure and powerful web server is required.

In this guide, we will demonstrate how to install and configure some components on Debian 9 to support and serve Django applications. We will be setting up a PostgreSQL database instead of using the default SQLite database. We will configure the Gunicorn application server to interface with our applications. We will then set up Nginx to reverse proxy to Gunicorn, giving us access to its security and performance features to serve our apps.

Prerequisites

In order to complete this guide, you should have a fresh Debian 9 server instance with a basic firewall and a non-root user with sudo privileges configured. You can learn how to set this up by running through our initial server setup guide.

We will be installing Django within a virtual environment. Installing Django into an environment specific to your project will allow your projects and their requirements to be handled separately.

Once we have our database and application up and running, we will install and configure the Gunicorn application server. This will serve as an interface to our application, translating client requests from HTTP to Python calls that our application can process. We will then set up Nginx in front of Gunicorn to take advantage of its high performance connection handling mechanisms and its easy-to-implement security features.

Let's get started.

Step 1 — Installing the Packages from the Debian Repositories

To begin the process, we'll download and install all of the items we need from the Debian repositories. We will use the Python package manager pip to install additional components a bit later.

We need to update the local apt package index and then download and install the packages. The packages we install depend on which version of Python your project will use.

If you are using Django with Python 3, type:

sudo apt update
sudo apt install python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx curl

Django 1.11 is the last release of Django that will support Python 2. If you are starting new projects, it is strongly recommended that you choose Python 3. If you still need to use Python 2, type:

sudo apt update
sudo apt install python-pip python-dev libpq-dev postgresql postgresql-contrib nginx curl

This will install pip, the Python development files needed to build Gunicorn later, the Postgres database system and the libraries needed to interact with it, and the Nginx web server.

Step 2 — Creating the PostgreSQL Database and User

We're going to jump right in and create a database and database user for our Django application.

By default, Postgres uses an authentication scheme called "peer authentication" for local connections. Basically, this means that if the user's operating system username matches a valid Postgres username, that user can login with no further authentication.

During the Postgres installation, an operating system user named postgres was created to correspond to the postgres PostgreSQL administrative user. We need to use this user to perform administrative tasks. We can use sudo and pass in the username with the -u option.

Log into an interactive Postgres session by typing:

sudo -u postgres psql

You will be given a PostgreSQL prompt where we can set up our requirements.

First, create a database for your project:

CREATE DATABASE myproject;

Note: Every Postgres statement must end with a semi-colon, so make sure that your command ends with one if you are experiencing issues.

Next, create a database user for our project. Make sure to select a secure password:

CREATE USER myprojectuser WITH PASSWORD 'password';

Afterwards, we'll modify a few of the connection parameters for the user we just created. This will speed up database operations so that the correct values do not have to be queried and set each time a connection is established.

We are setting the default encoding to UTF-8, which Django expects. We are also setting the default transaction isolation scheme to "read committed", which blocks reads from uncommitted transactions. Lastly, we are setting the timezone. By default, our Django projects will be set to use UTC. These are all recommendations from the Django project itself:

ALTER ROLE myprojectuser SET client_encoding TO 'utf8';
ALTER ROLE myprojectuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE myprojectuser SET timezone TO 'UTC';

Now, we can give our new user access to administer our new database:

GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

When you are finished, exit out of the PostgreSQL prompt by typing:

\q

Postgres is now set up so that Django can connect to and manage its database information.

Step 3 — Creating a Python Virtual Environment for your Project

Now that we have our database, we can begin getting the rest of our project requirements ready. We will be installing our Python requirements within a virtual environment for easier management.

To do this, we first need access to the virtualenv command. We can install this with pip.

If you are using Python 3, upgrade pip and install the package by typing:

sudo -H pip3 install --upgrade pip
sudo -H pip3 install virtualenv

If you are using Python 2, upgrade pip and install the package by typing:

sudo -H pip install --upgrade pip
sudo -H pip install virtualenv

With virtualenv installed, we can start forming our project. Create and move into a directory where we can keep our project files:

mkdir ~/myprojectdir
cd ~/myprojectdir

Within the project directory, create a Python virtual environment by typing:

virtualenv myprojectenv

Before we install our project's Python requirements, we need to activate the virtual environment. You can do that by typing:

source myprojectenv/bin/activate

Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (myprojectenv)user@host:~/myprojectdir$.

With your virtual environment active, install Django, Gunicorn, and the psycopg2 PostgreSQL adaptor with the local instance of pip:

pip install django gunicorn psycopg2-binary

You should now have all of the software needed to start a Django project.

Step 4 — Creating and Configuring a New Django Project

With our Python components installed, we can create the actual Django project files.

Creating the Django Project

Since we already have a project directory, we will tell Django to install the files here. It will create a second level directory with the actual code, which is normal, and place a management script in this directory. The key to this is that we are defining the directory explicitly instead of allowing Django to make decisions relative to our current directory:

django-admin.py startproject myproject ~/myprojectdir

At this point, your project directory (~/myprojectdir in our case) should have the following content:

~/myprojectdir/manage.py: A Django project management script.
~/myprojectdir/myproject/: The Django project package. This should contain the __init__.py, settings.py, urls.py, and wsgi.py files.
~/myprojectdir/myprojectenv/: The virtual environment directory we created earlier.

Adjusting the Project Settings

The first thing we should do with our newly created project files is adjust the settings. Open the settings file in your text editor:

nano ~/myprojectdir/myproject/settings.py

Start by locating the ALLOWED_HOSTS directive. This defines a list of the server's addresses or domain names may be used to connect to the Django instance. Any incoming requests with a Host header that is not in this list will raise an exception. Django requires that you set this to prevent a certain class of security vulnerability.

In the square brackets, list the IP addresses or domain names that are associated with your Django server. Each item should be listed in quotations with entries separated by a comma. If you wish requests for an entire domain and any subdomains, prepend a period to the beginning of the entry. In the snippet below, there are a few commented out examples used to demonstrate:

Note: Be sure to include localhost as one of the options since we will be proxying connections through a local Nginx instance.

~/myprojectdir/myproject/settings.py

. . .
# The simplest case: just add the domain name(s) and IP addresses of your Django server
# ALLOWED_HOSTS = [ 'example.com', '203.0.113.5']
# To respond to 'example.com' and any subdomains, start the domain with a dot
# ALLOWED_HOSTS = ['.example.com', '203.0.113.5']
ALLOWED_HOSTS = ['your_server_domain_or_IP', 'second_domain_or_IP', . . ., 'localhost']

Next, find the section that configures database access. It will start with DATABASES. The configuration in the file is for a SQLite database. We already created a PostgreSQL database for our project, so we need to adjust the settings.

Change the settings with your PostgreSQL database information. We tell Django to use the psycopg2 adaptor we installed with pip. We need to give the database name, the database username, the database user's password, and then specify that the database is located on the local computer. You can leave the PORT setting as an empty string:

~/myprojectdir/myproject/settings.py

. . .

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'myproject',
        'USER': 'myprojectuser',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '',
    }
}

. . .

Next, move down to the bottom of the file and add a setting indicating where the static files should be placed. This is necessary so that Nginx can handle requests for these items. The following line tells Django to place them in a directory called static in the base project directory:

~/myprojectdir/myproject/settings.py

. . .

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Save and close the file when you are finished.

Completing Initial Project Setup

Now, we can migrate the initial database schema to our PostgreSQL database using the management script:

~/myprojectdir/manage.py makemigrations
~/myprojectdir/manage.py migrate

Create an administrative user for the project by typing:

~/myprojectdir/manage.py createsuperuser

You will have to select a username, provide an email address, and choose and confirm a password.

We can collect all of the static content into the directory location we configured by typing:

~/myprojectdir/manage.py collectstatic

You will have to confirm the operation. The static files will then be placed in a directory called static within your project directory.

If you followed the initial server setup guide, you should have a UFW firewall protecting your server. In order to test the development server, we'll have to allow access to the port we'll be using.

Create an exception for port 8000 by typing:

sudo ufw allow 8000

Finally, you can test our your project by starting up the Django development server with this command:

~/myprojectdir/manage.py runserver 0.0.0.0:8000

In your web browser, visit your server's domain name or IP address followed by :8000:

http://server_domain_or_IP:8000

You should see the default Django index page:

If you append /admin to the end of the URL in the address bar, you will be prompted for the administrative username and password you created with the createsuperuser command:

After authenticating, you can access the default Django admin interface:

When you are finished exploring, hit CTRL-C in the terminal window to shut down the development server.

Testing Gunicorn's Ability to Serve the Project

The last thing we want to do before leaving our virtual environment is test Gunicorn to make sure that it can serve the application. We can do this by entering our project directory and using gunicorn to load the project's WSGI module:

cd ~/myprojectdir
gunicorn --bind 0.0.0.0:8000 myproject.wsgi

This will start Gunicorn on the same interface that the Django development server was running on. You can go back and test the app again.

Note: The admin interface will not have any of the styling applied since Gunicorn does not know how to find the static CSS content responsible for this.

We passed Gunicorn a module by specifying the relative directory path to Django's wsgi.py file, which is the entry point to our application, using Python's module syntax. Inside of this file, a function called application is defined, which is used to communicate with the application. To learn more about the WSGI specification, click here.

When you are finished testing, hit CTRL-C in the terminal window to stop Gunicorn.

We're now finished configuring our Django application. We can back out of our virtual environment by typing:

deactivate

The virtual environment indicator in your prompt will be removed.

Step 5 — Creating systemd Socket and Service Files for Gunicorn

We have tested that Gunicorn can interact with our Django application, but we should implement a more robust way of starting and stopping the application server. To accomplish this, we'll make systemd service and socket files.

The Gunicorn socket will be created at boot and will listen for connections. When a connection occurs, systemd will automatically start the Gunicorn process to handle the connection.

Start by creating and opening a systemd socket file for Gunicorn with sudo privileges:

sudo nano /etc/systemd/system/gunicorn.socket

Inside, we will create a [Unit] section to describe the socket, a [Socket] section to define the socket location, and an [Install] section to make sure the socket is created at the right time:

/etc/systemd/system/gunicorn.socket

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

Save and close the file when you are finished.

Next, create and open a systemd service file for Gunicorn with sudo privileges in your text editor. The service filename should match the socket filename with the exception of the extension:

sudo nano /etc/systemd/system/gunicorn.service

Start with the [Unit] section, which is used to specify metadata and dependencies. We'll put a description of our service here and tell the init system to only start this after the networking target has been reached. Because our service relies on the socket from the socket file, we need to include a Requires directive to indicate that relationship:

/etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

Next, we'll open up the [Service] section. We'll specify the user and group that we want to process to run under. We will give our regular user account ownership of the process since it owns all of the relevant files. We'll give group ownership to the www-data group so that Nginx can communicate easily with Gunicorn.

We'll then map out the working directory and specify the command to use to start the service. In this case, we'll have to specify the full path to the Gunicorn executable, which is installed within our virtual environment. We will bind the process to the Unix socket we created within the /run directory so that the process can communicate with Nginx. We log all data to standard output so that the journald process can collect the Gunicorn logs. We can also specify any optional Gunicorn tweaks here. For example, we specified 3 worker processes in this case:

/etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myprojectdir
ExecStart=/home/sammy/myprojectdir/myprojectenv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

Finally, we'll add an [Install] section. This will tell systemd what to link this service to if we enable it to start at boot. We want this service to start when the regular multi-user system is up and running:

/etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myprojectdir
ExecStart=/home/sammy/myprojectdir/myprojectenv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

[Install]
WantedBy=multi-user.target

With that, our systemd service file is complete. Save and close it now.

We can now start and enable the Gunicorn socket. This will create the socket file at /run/gunicorn.sock now and at boot. When a connection is made to that socket, systemd will automatically start the gunicorn.service to handle it:

sudo systemctl start gunicorn.socket
sudo systemctl enable gunicorn.socket

We can confirm that the operation was successful by checking for the socket file.

Step 6 — Checking for the Gunicorn Socket File

Check the status of the process to find out whether it was able to start:

sudo systemctl status gunicorn.socket

Next, check for the existence of the gunicorn.sock file within the /run directory:

file /run/gunicorn.sock

Output/run/gunicorn.sock: socket

If the systemctl status command indicated that an error occurred or if you do not find the gunicorn.sock file in the directory, it's an indication that the Gunicorn socket was not able to be created correctly. Check the Gunicorn socket's logs by typing:

sudo journalctl -u gunicorn.socket

Take another look at your /etc/systemd/system/gunicorn.socket file to fix any problems before continuing.

Step 7 — Testing Socket Activation

Currently, if you've only started the gunicorn.socket unit, the gunicorn.service will not be active yet since the socket has not yet received any connections. You can check this by typing:

sudo systemctl status gunicorn

Output● gunicorn.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

To test the socket activation mechanism, we can send a connection to the socket through curl by typing:

curl --unix-socket /run/gunicorn.sock localhost

You should see the HTML output from your application in the terminal. This indicates that Gunicorn was started and was able to serve your Django application. You can verify that the Gunicorn service is running by typing:

sudo systemctl status gunicorn

Output● gunicorn.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-07-09 20:00:40 UTC; 4s ago
 Main PID: 1157 (gunicorn)
    Tasks: 4 (limit: 1153)
   CGroup: /system.slice/gunicorn.service
           ├─1157 /home/sammy/myprojectdir/myprojectenv/bin/python3 /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
           ├─1178 /home/sammy/myprojectdir/myprojectenv/bin/python3 /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
           ├─1180 /home/sammy/myprojectdir/myprojectenv/bin/python3 /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
           └─1181 /home/sammy/myprojectdir/myprojectenv/bin/python3 /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application

Jul 09 20:00:40 django1 systemd[1]: Started gunicorn daemon.
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1157] [INFO] Starting gunicorn 19.9.0
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1157] [INFO] Listening at: unix:/run/gunicorn.sock (1157)
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1157] [INFO] Using worker: sync
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1178] [INFO] Booting worker with pid: 1178
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1180] [INFO] Booting worker with pid: 1180
Jul 09 20:00:40 django1 gunicorn[1157]: [2018-07-09 20:00:40 +0000] [1181] [INFO] Booting worker with pid: 1181
Jul 09 20:00:41 django1 gunicorn[1157]:  - - [09/Jul/2018:20:00:41 +0000] "GET / HTTP/1.1" 200 16348 "-" "curl/7.58.0"

If the output from curl or the output of systemctl status indicates that a problem occurred, check the logs for additional details:

sudo journalctl -u gunicorn

Check your /etc/systemd/system/gunicorn.service file for problems. If you make changes to the /etc/systemd/system/gunicorn.service file, reload the daemon to reread the service definition and restart the Gunicorn process by typing:

sudo systemctl daemon-reload
sudo systemctl restart gunicorn

Make sure you troubleshoot the above issues before continuing.

Step 8 — Configure Nginx to Proxy Pass to Gunicorn

Now that Gunicorn is set up, we need to configure Nginx to pass traffic to the process.

Start by creating and opening a new server block in Nginx's sites-available directory:

sudo nano /etc/nginx/sites-available/myproject

Inside, open up a new server block. We will start by specifying that this block should listen on the normal port 80 and that it should respond to our server's domain name or IP address:

/etc/nginx/sites-available/myproject

server {
    listen 80;
    server_name server_domain_or_IP;
}

Next, we will tell Nginx to ignore any problems with finding a favicon. We will also tell it where to find the static assets that we collected in our ~/myprojectdir/static directory. All of these files have a standard URI prefix of "/static", so we can create a location block to match those requests:

/etc/nginx/sites-available/myproject

server {
    listen 80;
    server_name server_domain_or_IP;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/sammy/myprojectdir;
    }
}

Finally, we'll create a location / {} block to match all other requests. Inside of this location, we'll include the standard proxy_params file included with the Nginx installation and then we will pass the traffic directly to the Gunicorn socket:

/etc/nginx/sites-available/myproject

server {
    listen 80;
    server_name server_domain_or_IP;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/sammy/myprojectdir;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/run/gunicorn.sock;
    }
}

Save and close the file when you are finished. Now, we can enable the file by linking it to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

Test your Nginx configuration for syntax errors by typing:

sudo nginx -t

If no errors are reported, go ahead and restart Nginx by typing:

sudo systemctl restart nginx

Finally, we need to open up our firewall to normal traffic on port 80. Since we no longer need access to the development server, we can remove the rule to open port 8000 as well:

sudo ufw delete allow 8000
sudo ufw allow 'Nginx Full'

You should now be able to go to your server's domain or IP address to view your application.

Note: After configuring Nginx, the next step should be securing traffic to the server using SSL/TLS. This is important because without it, all information, including passwords are sent over the network in plain text.

If you have a domain name, the easiest way get an SSL certificate to secure your traffic is using Let's Encrypt. Follow this guide to set up Let's Encrypt with Nginx on Debian 9. Follow the procedure using the Nginx server block we created in this guide.

If you do not have a domain name, you can still secure your site for testing and learning with a self-signed SSL certificate. Again, follow the process using the Nginx server block we created in this tutorial.

Troubleshooting Nginx and Gunicorn

If this last step does not show your application, you will need to troubleshoot your installation.

Nginx Is Showing the Default Page Instead of the Django Application

If Nginx displays the default page instead of proxying to your application, it usually means that you need to adjust the server_name within the /etc/nginx/sites-available/myproject file to point to your server's IP address or domain name.

Nginx uses the server_name to determine which server block to use to respond to requests. If you are seeing the default Nginx page, it is a sign that Nginx wasn't able to match the request to a sever block explicitly, so it's falling back on the default block defined in /etc/nginx/sites-available/default.

The server_name in your project's server block must be more specific than the one in the default server block to be selected.

Nginx Is Displaying a 502 Bad Gateway Error Instead of the Django Application

A 502 error indicates that Nginx is unable to successfully proxy the request. A wide range of configuration problems express themselves with a 502 error, so more information is required to troubleshoot properly.

The primary place to look for more information is in Nginx's error logs. Generally, this will tell you what conditions caused problems during the proxying event. Follow the Nginx error logs by typing:

sudo tail -F /var/log/nginx/error.log

Now, make another request in your browser to generate a fresh error (try refreshing the page). You should see a fresh error message written to the log. If you look at the message, it should help you narrow down the problem.

You might see some of the following message:

connect() to unix:/run/gunicorn.sock failed (2: No such file or directory)

This indicates that Nginx was unable to find the gunicorn.sock file at the given location. You should compare the proxy_pass location defined within /etc/nginx/sites-available/myproject file to the actual location of the gunicorn.sock file generated by the gunicorn.socket systemd unit.

If you cannot find a gunicorn.sock file within the /run directory, it generally means that the systemd socket file was unable to create it. Go back to the section on checking for the Gunicorn socket file to step through the troubleshooting steps for Gunicorn.

connect() to unix:/run/gunicorn.sock failed (13: Permission denied)

This indicates that Nginx was unable to connect to the Gunicorn socket because of permissions problems. This can happen when the procedure is followed using the root user instead of a sudo user. While systemd is able to create the Gunicorn socket file, Nginx is unable to access it.

This can happen if there are limited permissions at any point between the root directory (/) the gunicorn.sock file. We can see the permissions and ownership values of the socket file and each of its parent directories by passing the absolute path to our socket file to the namei command:

namei -l /run/gunicorn.sock

Outputf: /run/gunicorn.sock
drwxr-xr-x root root /
drwxr-xr-x root root run
srw-rw-rw- root root gunicorn.sock

The output displays the permissions of each of the directory components. By looking at the permissions (first column), owner (second column) and group owner (third column), we can figure out what type of access is allowed to the socket file.

In the above example, the socket file and each of the directories leading up to the socket file have world read and execute permissions (the permissions column for the directories end with r-x instead of ---). The Nginx process should be able to access the socket successfully.

If any of the directories leading up to the socket do not have world read and execute permission, Nginx will not be able to access the socket without allowing world read and execute permissions or making sure group ownership is given to a group that Nginx is a part of.

Django Is Displaying: "could not connect to server: Connection refused"

One message that you may see from Django when attempting to access parts of the application in the web browser is:

OperationalError at /admin/login/
could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5432?

This indicates that Django is unable to connect to the Postgres database. Make sure that the Postgres instance is running by typing:

sudo systemctl status postgresql

If it is not, you can start it and enable it to start automatically at boot (if it is not already configured to do so) by typing:

sudo systemctl start postgresql
sudo systemctl enable postgresql

If you are still having issues, make sure the database settings defined in the ~/myprojectdir/myproject/settings.py file are correct.

Further Troubleshooting

For additional troubleshooting, the logs can help narrow down root causes. Check each of them in turn and look for messages indicating problem areas.

The following logs may be helpful:

Check the Nginx process logs by typing: sudo journalctl -u nginx
Check the Nginx access logs by typing: sudo less /var/log/nginx/access.log
Check the Nginx error logs by typing: sudo less /var/log/nginx/error.log
Check the Gunicorn application logs by typing: sudo journalctl -u gunicorn
Check the Gunicorn socket logs by typing: sudo journalctl -u gunicorn.socket

As you update your configuration or application, you will likely need to restart the processes to adjust to your changes.

If you update your Django application, you can restart the Gunicorn process to pick up the changes by typing:

sudo systemctl restart gunicorn

If you change Gunicorn socket or service files, reload the daemon and restart the process by typing:

sudo systemctl daemon-reload
sudo systemctl restart gunicorn.socket gunicorn.service

If you change the Nginx server block configuration, test the configuration and then Nginx by typing:

sudo nginx -t && sudo systemctl restart nginx

These commands are helpful for picking up changes as you adjust your configuration.

Conclusion

In this guide, we've set up a Django project in its own virtual environment. We've configured Gunicorn to translate client requests so that Django can handle them. Afterwards, we set up Nginx to act as a reverse proxy to handle client connections and serve the correct project depending on the client request.

Django makes creating projects and applications simple by providing many of the common pieces, allowing you to focus on the unique elements. By leveraging the general tool chain described in this article, you can easily serve the applications you create from a single server.

How To Configure BIND as a Private Network DNS Server on Debian 9

2018-09-06T19:18:17Z

Introduction

An important part of managing server configuration and infrastructure includes maintaining an easy way to look up network interfaces and IP addresses by name, by setting up a proper Domain Name System (DNS). Using fully qualified domain names (FQDNs), instead of IP addresses, to specify network addresses eases the configuration of services and applications, and increases the maintainability of configuration files. Setting up your own DNS for your private network is a great way to improve the management of your servers.

In this tutorial, we will go over how to set up an internal DNS server, using the BIND name server software (BIND9) on Debian 9, that can be used by your servers to resolve private hostnames and private IP addresses. This provides a central way to manage your internal hostnames and private IP addresses, which is indispensable when your environment expands to more than a few hosts.

Prerequisites

To complete this tutorial, you will need the following infrastructure. Create each server in the same datacenter with private networking enabled:

A fresh Debian 9 server to serve as the Primary DNS server, ns1
(Recommended) A second Debian 9 server to serve as a Secondary DNS server, ns2
Additional servers in the same datacenter that will be using your DNS servers

On each of these servers, configure administrative access via a sudo user and a firewall by following our Debian 9 initial server setup guide.

If you are unfamiliar with DNS concepts, it is recommended that you read at least the first three parts of our Introduction to Managing DNS.

Example Infrastructure and Goals

For the purposes of this article, we will assume the following:

We have two servers which will be designated as our DNS name servers. We will refer to these as ns1 and ns2 in this guide.
We have two additional client servers that will be using the DNS infrastructure we create. We will call these host1 and host2 in this guide. You can add as many as you'd like for your infrastructure.
All of these servers exist in the same datacenter. We will assume that this is the nyc3 datacenter.
All of these servers have private networking enabled (and are on the 10.128.0.0/16 subnet. You will likely have to adjust this for your servers).
All servers are connected to a project that runs on "example.com". Since our DNS system will be entirely internal and private, you do not have to purchase a domain name. However, using a domain you own may help avoid conflicts with publicly routable domains.

With these assumptions, we decide that it makes sense to use a naming scheme that uses "nyc3.example.com" to refer to our private subnet or zone. Therefore, host1's private Fully-Qualified Domain Name (FQDN) will be host1.nyc3.example.com. Refer to the following table the relevant details:

Host	Role	Private FQDN	Private IP Address
ns1	Primary DNS Server	ns1.nyc3.example.com	10.128.10.11
ns2	Secondary DNS Server	ns2.nyc3.example.com	10.128.20.12
host1	Generic Host 1	host1.nyc3.example.com	10.128.100.101
host2	Generic Host 2	host2.nyc3.example.com	10.128.200.102

Note

Your existing setup will be different, but the example names and IP addresses will be used to demonstrate how to configure a DNS server to provide a functioning internal DNS. You should be able to easily adapt this setup to your own environment by replacing the host names and private IP addresses with your own. It is not necessary to use the region name of the datacenter in your naming scheme, but we use it here to denote that these hosts belong to a particular datacenter's private network. If you utilize multiple datacenters, you can set up an internal DNS within each respective datacenter.

By the end of this tutorial, we will have a primary DNS server, ns1, and optionally a secondary DNS server, ns2, which will serve as a backup.

Let's get started by installing our Primary DNS server, ns1.

Installing BIND on DNS Servers

Note

Text that is highlighted in red is important! It will often be used to denote something that needs to be replaced with your own settings or that it should be modified or added to a configuration file. For example, if you see something like host1.nyc3.example.com, replace it with the FQDN of your own server. Likewise, if you see host1_private_IP, replace it with the private IP address of your own server.

On both DNS servers, ns1 and ns2, update the apt package cache by typing:

sudo apt update

Now install BIND:

sudo apt install bind9 bind9utils bind9-doc

Setting Bind to IPv4 Mode

Before continuing, let's set BIND to IPv4 mode since our private networking uses IPv4 exclusively. On both servers, edit the bind9 default settings file by typing:

sudo nano /etc/default/bind9

Add "-4" to the end of the OPTIONS parameter. It should look like the following:

/etc/default/bind9

. . .
OPTIONS="-u bind -4"

Save and close the file when you are finished.

Restart BIND to implement the changes:

sudo systemctl restart bind9

Now that BIND is installed, let's configure the primary DNS server.

Configuring the Primary DNS Server

BIND's configuration consists of multiple files, which are included from the main configuration file, named.conf. These filenames begin with named because that is the name of the process that BIND runs (short for "domain name daemon"). We will start with configuring the options file.

Configuring the Options File

On ns1, open the named.conf.options file for editing:

sudo nano /etc/bind/named.conf.options

Above the existing options block, create a new ACL (access control list) block called "trusted". This is where we will define a list of clients that we will allow recursive DNS queries from (i.e. your servers that are in the same datacenter as ns1). Using our example private IP addresses, we will add ns1, ns2, host1, and host2 to our list of trusted clients:

/etc/bind/named.conf.options — 1 of 3

acl "trusted" {
        10.128.10.11;    # ns1 - can be set to localhost
        10.128.20.12;    # ns2
        10.128.100.101;  # host1
        10.128.200.102;  # host2
};

options {

        . . .

Now that we have our list of trusted DNS clients, we will want to edit the options block. Currently, the start of the block looks like the following:

/etc/bind/named.conf.options — 2 of 3

        . . .
};

options {
        directory "/var/cache/bind";
        . . .
}

Below the directory directive, add the highlighted configuration lines (and substitute in the proper ns1 IP address) so it looks something like this:

/etc/bind/named.conf.options — 3 of 3

        . . .

};

options {
        directory "/var/cache/bind";

        recursion yes;                 # enables resursive queries
        allow-recursion { trusted; };  # allows recursive queries from "trusted" clients
        listen-on { 10.128.10.11; };   # ns1 private IP address - listen on private network only
        allow-transfer { none; };      # disable zone transfers by default

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };

        . . .
};

When you are finished, save and close the named.conf.options file. The above configuration specifies that only your own servers (the "trusted" ones) will be able to query your DNS server for outside domains.

Next, we will configure the local file, to specify our DNS zones.

Configuring the Local File

On ns1, open the named.conf.local file for editing:

sudo nano /etc/bind/named.conf.local

Aside from a few comments, the file should be empty. Here, we will specify our forward and reverse zones. DNS zones designate a specific scope for managing and defining DNS records. Since our domains will all be within the "nyc3.example.com" subdomain, we will use that as our forward zone. Because our servers' private IP addresses are each in the 10.128.0.0/16 IP space, we will set up a reverse zone so that we can define reverse lookups within that range.

Add the forward zone with the following lines, substituting the zone name with your own and the secondary DNS server's private IP address in the allow-transfer directive:

/etc/bind/named.conf.local — 1 of 2

zone "nyc3.example.com" {
    type master;
    file "/etc/bind/zones/db.nyc3.example.com"; # zone file path
    allow-transfer { 10.128.20.12; };           # ns2 private IP address - secondary
};

Assuming that our private subnet is 10.128.0.0/16, add the reverse zone by with the following lines (note that our reverse zone name starts with "128.10" which is the octet reversal of "10.128"):

/etc/bind/named.conf.local — 2 of 2

    . . .
};

zone "128.10.in-addr.arpa" {
    type master;
    file "/etc/bind/zones/db.10.128";  # 10.128.0.0/16 subnet
    allow-transfer { 10.128.20.12; };  # ns2 private IP address - secondary
};

If your servers span multiple private subnets but are in the same datacenter, be sure to specify an additional zone and zone file for each distinct subnet. When you are finished adding all of your desired zones, save and exit the named.conf.local file.

Now that our zones are specified in BIND, we need to create the corresponding forward and reverse zone files.

Creating the Forward Zone File

The forward zone file is where we define DNS records for forward DNS lookups. That is, when the DNS receives a name query, "host1.nyc3.example.com" for example, it will look in the forward zone file to resolve host1's corresponding private IP address.

Let's create the directory where our zone files will reside. According to our named.conf.local configuration, that location should be /etc/bind/zones:

sudo mkdir /etc/bind/zones

We will base our forward zone file on the sample db.local zone file. Copy it to the proper location with the following commands:

sudo cp /etc/bind/db.local /etc/bind/zones/db.nyc3.example.com

Now let's edit our forward zone file:

sudo nano /etc/bind/zones/db.nyc3.example.com

Initially, it will look something like the following:

/etc/bind/zones/db.nyc3.example.com — original

$TTL    604800
@       IN      SOA     localhost. root.localhost. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      localhost.      ; delete this line
@       IN      A       127.0.0.1       ; delete this line
@       IN      AAAA    ::1             ; delete this line

First, you will want to edit the SOA record. Replace the first "localhost" with ns1's FQDN, then replace "root.localhost" with "admin.nyc3.example.com". Every time you edit a zone file, you need to increment the serial value before you restart the named process. We will increment it to "3". It should now look something like this:

/etc/bind/zones/db.nyc3.example.com — updated 1 of 3

@       IN      SOA     ns1.nyc3.example.com. admin.nyc3.example.com. (
                              3         ; Serial

                              . . .

Next, delete the three records at the end of the file (after the SOA record). If you're not sure which lines to delete, they are marked with a "delete this line" comment above.

At the end of the file, add your name server records with the following lines (replace the names with your own). Note that the second column specifies that these are "NS" records:

/etc/bind/zones/db.nyc3.example.com — updated 2 of 3

. . .

; name servers - NS records
    IN      NS      ns1.nyc3.example.com.
    IN      NS      ns2.nyc3.example.com.

Now, add the A records for your hosts that belong in this zone. This includes any server whose name we want to end with ".nyc3.example.com" (substitute the names and private IP addresses). Using our example names and private IP addresses, we will add A records for ns1, ns2, host1, and host2 like so:

/etc/bind/zones/db.nyc3.example.com — updated 3 of 3

. . .

; name servers - A records
ns1.nyc3.example.com.          IN      A       10.128.10.11
ns2.nyc3.example.com.          IN      A       10.128.20.12

; 10.128.0.0/16 - A records
host1.nyc3.example.com.        IN      A      10.128.100.101
host2.nyc3.example.com.        IN      A      10.128.200.102

Save and close the db.nyc3.example.com file.

Our final example forward zone file looks like the following:

/etc/bind/zones/db.nyc3.example.com — updated

$TTL    604800
@       IN      SOA     ns1.nyc3.example.com. admin.nyc3.example.com. (
                  3     ; Serial
             604800     ; Refresh
              86400     ; Retry
            2419200     ; Expire
             604800 )   ; Negative Cache TTL
;
; name servers - NS records
     IN      NS      ns1.nyc3.example.com.
     IN      NS      ns2.nyc3.example.com.

; name servers - A records
ns1.nyc3.example.com.          IN      A       10.128.10.11
ns2.nyc3.example.com.          IN      A       10.128.20.12

; 10.128.0.0/16 - A records
host1.nyc3.example.com.        IN      A      10.128.100.101
host2.nyc3.example.com.        IN      A      10.128.200.102

Now let's move onto the reverse zone file(s).

Creating the Reverse Zone File(s)

Reverse zone files are where we define DNS PTR records for reverse DNS lookups. That is, when the DNS receives a query by IP address, "10.128.100.101" for example, it will look in the reverse zone file(s) to resolve the corresponding FQDN, "host1.nyc3.example.com" in this case.

On ns1, for each reverse zone specified in the named.conf.local file, create a reverse zone file. We will base our reverse zone file(s) on the sample db.127 zone file. Copy it to the proper location with the following commands (substituting the destination filename so it matches your reverse zone definition):

sudo cp /etc/bind/db.127 /etc/bind/zones/db.10.128

Edit the reverse zone file that corresponds to the reverse zone(s) defined in named.conf.local:

sudo nano /etc/bind/zones/db.10.128

Initially, it will look something like the following:

/etc/bind/zones/db.10.128 — original

$TTL    604800
@       IN      SOA     localhost. root.localhost. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      localhost.      ; delete this line
1.0.0   IN      PTR     localhost.      ; delete this line

In the same manner as the forward zone file, you will want to edit the SOA record and increment the serial value. It should look something like this:

/etc/bind/zones/db.10.128 — updated 1 of 3

@       IN      SOA     ns1.nyc3.example.com. admin.nyc3.example.com. (
                              3         ; Serial

                              . . .

Now delete the two records at the end of the file (after the SOA record). If you're not sure which lines to delete, they are marked with a "delete this line" comment above.

At the end of the file, add your name server records with the following lines (replace the names with your own). Note that the second column specifies that these are "NS" records:

/etc/bind/zones/db.10.128 — updated 2 of 3

. . .

; name servers - NS records
      IN      NS      ns1.nyc3.example.com.
      IN      NS      ns2.nyc3.example.com.

Then add PTR records for all of your servers whose IP addresses are on the subnet of the zone file that you are editing. In our example, this includes all of our hosts because they are all on the 10.128.0.0/16 subnet. Note that the first column consists of the last two octets of your servers' private IP addresses in reversed order. Be sure to substitute names and private IP addresses to match your servers:

/etc/bind/zones/db.10.128 — updated 3 of 3

. . .

; PTR Records
11.10   IN      PTR     ns1.nyc3.example.com.    ; 10.128.10.11
12.20   IN      PTR     ns2.nyc3.example.com.    ; 10.128.20.12
101.100 IN      PTR     host1.nyc3.example.com.  ; 10.128.100.101
102.200 IN      PTR     host2.nyc3.example.com.  ; 10.128.200.102

Save and close the reverse zone file (repeat this section if you need to add more reverse zone files).

Our final example reverse zone file looks like the following:

/etc/bind/zones/db.10.128 — updated

$TTL    604800
@       IN      SOA     nyc3.example.com. admin.nyc3.example.com. (
                              3         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
; name servers
      IN      NS      ns1.nyc3.example.com.
      IN      NS      ns2.nyc3.example.com.

; PTR Records
11.10   IN      PTR     ns1.nyc3.example.com.    ; 10.128.10.11
12.20   IN      PTR     ns2.nyc3.example.com.    ; 10.128.20.12
101.100 IN      PTR     host1.nyc3.example.com.  ; 10.128.100.101
102.200 IN      PTR     host2.nyc3.example.com.  ; 10.128.200.102

We're done editing our files, so next we can check our files for errors.

Checking the BIND Configuration Syntax

Run the following command to check the syntax of the named.conf* files:

sudo named-checkconf

If your named configuration files have no syntax errors, you will return to your shell prompt and see no error messages. If there are problems with your configuration files, review the error message and the "Configure Primary DNS Server" section, then try named-checkconf again.

The named-checkzone command can be used to check the correctness of your zone files. Its first argument specifies a zone name, and the second argument specifies the corresponding zone file, which are both defined in named.conf.local.

For example, to check the "nyc3.example.com" forward zone configuration, run the following command (change the names to match your forward zone and file):

sudo named-checkzone nyc3.example.com /etc/bind/zones/db.nyc3.example.com

And to check the "128.10.in-addr.arpa" reverse zone configuration, run the following command (change the numbers to match your reverse zone and file):

sudo named-checkzone 128.10.in-addr.arpa /etc/bind/zones/db.10.128

When all of your configuration and zone files have no errors in them, you should be ready to restart the BIND service.

Restarting BIND

Restart BIND:

sudo systemctl restart bind9

If you have the UFW firewall configured, open up access to BIND by typing:

sudo ufw allow Bind9

Your primary DNS server is now setup and ready to respond to DNS queries. Let's move on to creating the secondary DNS server.

Configuring the Secondary DNS Server

In most environments, it is a good idea to set up a secondary DNS server that will respond to requests if the primary becomes unavailable. Luckily, the secondary DNS server is much easier to configure.

On ns2, edit the named.conf.options file:

sudo nano /etc/bind/named.conf.options

At the top of the file, add the ACL with the private IP addresses of all of your trusted servers:

/etc/bind/named.conf.options — updated 1 of 2 (secondary)

acl "trusted" {
        10.128.10.11;   # ns1
        10.128.20.12;   # ns2 - can be set to localhost
        10.128.100.101;  # host1
        10.128.200.102;  # host2
};

options {

        . . .

Below the directory directive, add the following lines:

/etc/bind/named.conf.options — updated 2 of 2 (secondary)

        recursion yes;
        allow-recursion { trusted; };
        listen-on { 10.128.20.12; };      # ns2 private IP address
        allow-transfer { none; };          # disable zone transfers by default

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };

Save and close the named.conf.options file. This file should look exactly like ns1's named.conf.options file except it should be configured to listen on ns2's private IP address.

Now edit the named.conf.local file:

sudo nano /etc/bind/named.conf.local

Define slave zones that correspond to the master zones on the primary DNS server. Note that the type is "slave", the file does not contain a path, and there is a masters directive which should be set to the primary DNS server's private IP address. If you defined multiple reverse zones in the primary DNS server, make sure to add them all here:

/etc/bind/named.conf.local — updated (secondary)

zone "nyc3.example.com" {
    type slave;
    file "db.nyc3.example.com";
    masters { 10.128.10.11; };  # ns1 private IP
};

zone "128.10.in-addr.arpa" {
    type slave;
    file "db.10.128";
    masters { 10.128.10.11; };  # ns1 private IP
};

Now save and close the named.conf.local file.

Run the following command to check the validity of your configuration files:

sudo named-checkconf

Once that checks out, restart BIND:

sudo systemctl restart bind9

Allow DNS connections to the server by altering the UFW firewall rules:

sudo ufw allow Bind9

Now you have primary and secondary DNS servers for private network name and IP address resolution. Now you must configure your client servers to use your private DNS servers.

Configuring DNS Clients

Before all of your servers in the "trusted" ACL can query your DNS servers, you must configure each of them to use ns1 and ns2 as name servers. This process varies depending on OS, but for most Linux distributions it involves adding your name servers to the /etc/resolv.conf file.

Ubuntu 18.04 Clients

On Ubuntu 18.04, networking is configured with Netplan, an abstraction that allows you to write standardized network configuration and apply it to incompatible backend networking software. To configure DNS, we need to write a Netplan configuration file.

First, find the device associated with your private network by querying the private subnet with the ip address command:

ip address show to 10.128.0.0/16

Output
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 10.128.100.101/16 brd 10.128.255.255 scope global eth1
           valid_lft forever preferred_lft forever

In this example, the private interface is eth1.

Next, create a new file in /etc/netplan called 00-private-nameservers.yaml:

sudo nano /etc/netplan/00-private-nameservers.yaml

Inside, paste the following contents. You will need to modify the interface of the private network, the addresses of your ns1 and ns2 DNS servers, and the DNS zone:

Note: Netplan uses the YAML data serialization format for its configuration files. Because YAML uses indentation and whitespace to define its data structure, make sure that your definition uses consistent indentation to avoid errors.

/etc/netplan 00-private-nameservers.yaml

network:
    version: 2
    ethernets:
        eth1:                                 # Private network interface
            nameservers:
                addresses:
                - 10.128.10.11                # Private IP for ns1
                - 10.132.20.12                # Private IP for ns2
                search: [ nyc3.example.com ]  # DNS zone

Save and close the file when you are finished.

Next, tell Netplan to attempt to use the new configuration file by using netplan try. If there are problems that cause a loss of networking, Netplan will automatically roll back the changes after a timeout:

sudo netplan try

OutputWarning: Stopping systemd-networkd.service, but it can still be activated by:
  systemd-networkd.socket
Do you want to keep these settings?


Press ENTER before the timeout to accept the new configuration


Changes will revert in 120 seconds

If the countdown is updating correctly at the bottom, the new configuration is at least functional enough to not break your SSH connection. Press ENTER to accept the new configuration.

Now, check that the system's DNS resolver to determine if your DNS configuration has been applied:

sudo systemd-resolve --status

Scroll down until you see the section for your private network interface. You should see the private IP addresses for your DNS servers listed first, followed by some fallback values. Your domain should should be in the "DNS Domain":

Output. . .
Link 3 (eth1)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.128.10.11
                      10.128.20.12
                      67.207.67.2
                      67.207.67.3
          DNS Domain: nyc3.example.com
. . .

Your client should now be configured to use your internal DNS servers.

Ubuntu 16.04 and Debian Clients

On Ubuntu 16.04 and Debian Linux servers, you can edit the /etc/network/interfaces file:

sudo nano /etc/network/interfaces

Inside, find the dns-nameservers line. If it is attached to the lo interface, move it to your networking interface (eth0 or eth1 for example). Next, prepend your own name servers in front of the list that is currently there. Below that line, add a dns-search option pointed to the base domain of your infrastructure. In our case, this would be "nyc3.example.com":

/etc/network/interfaces

    . . .

    dns-nameservers 10.128.10.11 10.128.20.12 8.8.8.8
    dns-search nyc3.example.com

    . . .

Save and close the file when you are finished.

Make sure that the resolvconf package is installed on your system:

sudo apt update
sudo apt install resolvconf

Now, restart your networking services, applying the new changes with the following commands. Make sure you replace eth0 with the name of your networking interface:

sudo ifdown --force eth0 && sudo ip addr flush dev eth0 && sudo ifup --force eth0

This should restart your network without dropping your current connection. If it worked correctly, you should see something like this:

OutputRTNETLINK answers: No such process
Waiting for DAD... Done

Double check that your settings were applied by typing:

cat /etc/resolv.conf

You should see your name servers in the /etc/resolv.conf file, as well as your search domain:

Output# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.128.10.11
nameserver 10.128.20.12
nameserver 8.8.8.8
search nyc3.example.com

Your client is now configured to use your DNS servers.

CentOS Clients

On CentOS, RedHat, and Fedora Linux, edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file. You may have to substitute eth0 with the name of your primary network interface:

sudo nano /etc/sysconfig/network-scripts/ifcfg-eth0

Search for the DNS1 and DNS2 options and set them to the private IP addresses of your primary and secondary name servers. Add a DOMAIN parameter that with your infrastructure's base domain. In this guide, that would be "nyc3.example.com":

/etc/sysconfig/network-scripts/ifcfg-eth0

. . .
DNS1=10.128.10.11
DNS2=10.128.20.12
DOMAIN='nyc3.example.com'
. . .

Save and close the file when you are finished.

Now, restart the networking service by typing:

sudo systemctl restart network

The command may hang for a few seconds, but should return you to the prompt shortly.

Check that your changes were applied by typing:

cat /etc/resolv.conf

You should see your name servers and search domain in the list:

/etc/resolv.conf

nameserver 10.128.10.11
nameserver 10.128.20.12
search nyc3.example.com

Your client should now be able to connect to and use your DNS servers.

Testing Clients

Use nslookup to test if your clients can query your name servers. You should be able to do this on all of the clients that you have configured and are in the "trusted" ACL.

For CentOS clients, you may need to install the utility with:

sudo yum install bind-utils

For Debian clients, you can install with:

sudo apt install dnsutils

We can start by performing a forward lookup.

Forward Lookup

For example, we can perform a forward lookup to retrieve the IP address of host1.nyc3.example.com by running the following command:

nslookup host1

Querying "host1" expands to "host1.nyc3.example.com because of the search option is set to your private subdomain, and DNS queries will attempt to look on that subdomain before looking for the host elsewhere. The output of the command above would look like the following:

OutputServer:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   host1.nyc3.example.com
Address: 10.128.100.101

Next, we can check reverse lookups.

Reverse Lookup

To test the reverse lookup, query the DNS server with host1's private IP address:

nslookup 10.128.100.101

You should see output that looks like the following:

Output11.10.128.10.in-addr.arpa   name = host1.nyc3.example.com.

Authoritative answers can be found from:

If all of the names and IP addresses resolve to the correct values, that means that your zone files are configured properly. If you receive unexpected values, be sure to review the zone files on your primary DNS server (e.g. db.nyc3.example.com and db.10.128).

Congratulations! Your internal DNS servers are now set up properly! Now we will cover maintaining your zone records.

Maintaining DNS Records

Now that you have a working internal DNS, you need to maintain your DNS records so they accurately reflect your server environment.

Adding a Host to DNS

Whenever you add a host to your environment (in the same datacenter), you will want to add it to DNS. Here is a list of steps that you need to take:

Primary Name Server

Forward zone file: Add an "A" record for the new host, increment the value of "Serial"
Reverse zone file: Add a "PTR" record for the new host, increment the value of "Serial"
Add your new host's private IP address to the "trusted" ACL (named.conf.options)

Test your configuration files:

sudo named-checkconf
sudo named-checkzone nyc3.example.com db.nyc3.example.com
sudo named-checkzone 128.10.in-addr.arpa /etc/bind/zones/db.10.128

Then reload BIND:

sudo systemctl reload bind9

Your primary server should be configured for the new host now.

Secondary Name Server

Add your new host's private IP address to the "trusted" ACL (named.conf.options)

Check the configuration syntax:

sudo named-checkconf

Then reload BIND:

sudo systemctl reload bind9

Your secondary server will now accept connections from the new host.

Configure New Host to Use Your DNS

Configure /etc/resolv.conf to use your DNS servers
Test using nslookup

Removing Host from DNS

If you remove a host from your environment or want to just take it out of DNS, just remove all the things that were added when you added the server to DNS (i.e. the reverse of the steps above).

Conclusion

Now you may refer to your servers' private network interfaces by name, rather than by IP address. This makes configuration of services and applications easier because you no longer have to remember the private IP addresses, and the files will be easier to read and understand. Also, now you can change your configurations to point to a new servers in a single place, your primary DNS server, instead of having to edit a variety of distributed configuration files, which eases maintenance.

Once you have your internal DNS set up, and your configuration files are using private FQDNs to specify network connections, it is critical that your DNS servers are properly maintained. If they both become unavailable, your services and applications that rely on them will cease to function properly. This is why it is recommended to set up your DNS with at least one secondary server, and to maintain working backups of all of them.

How To Install the Anaconda Python Distribution on Debian 9

2018-09-06T18:37:52Z

Introduction

Anaconda is an open-source package manager, environment manager, and distribution of the Python and R programming languages. Designed for data science and machine learning workflows, it is commonly used for large-scale data processing, scientific computing, and predictive analytics.

Available in both free and paid enterprise versions, Anaconda offers a collection of over 1,000 data science packages. The Anaconda distribution ships with the conda command-line utility. You can learn more about Anaconda and conda by reading the official Anaconda Documentation.

This tutorial will guide you through installing the Python 3 version of Anaconda on a Debian 9 server.

Prerequisites

Before you begin with this guide, you should have a non-root user with sudo privileges set up on your server.

You can achieve this prerequisite by completing our Debian 9 initial server setup guide.

Installing Anaconda

The best way to install Anaconda is to download the latest Anaconda installer bash script, verify it, and then run it.

Find the latest version of Anaconda for Python 3 at the Downloads page accessible via the Anaconda home page. At the time of writing, the latest version is 5.2, but you should use a later stable version if it is available.

Next, change to the /tmp directory on your server. This is a good directory to download ephemeral items, like the Anaconda bash script, which we won't need after running it.

cd /tmp

We’ll use the curl command-line tool to download the script. Install curl:

sudo apt install curl

Now, use curl to download the link that you copied from the Anaconda website:

curl -O https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh

We can now verify the data integrity of the installer with cryptographic hash verification through the SHA-256 checksum. We’ll use the sha256sum command along with the filename of the script:

sha256sum Anaconda3-5.2.0-Linux-x86_64.sh

You’ll receive output that looks similar to this:

Output
09f53738b0cd3bb96f5b1bac488e5528df9906be2480fe61df40e0e0d19e3d48  Anaconda3-5.2.0-Linux-x86_64.sh

You should check the output against the hashes available at the Anaconda with Python 3 on 64-bit Linux page for your appropriate Anaconda version. As long as your output matches the hash displayed in the sha2561 row, you’re good to go.

Now we can run the script:

bash Anaconda3-5.2.0-Linux-x86_64.sh

You’ll receive the following output:

Output
Welcome to Anaconda3 5.2.0

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>

Press ENTER to continue and then press ENTER to read through the license. Once you’re done reading the license, you’ll be prompted to approve the license terms:

OutputDo you approve the license terms? [yes|no]

As long as you agree, type yes.

At this point, you’ll be prompted to choose the location of the installation. You can press ENTER to accept the default location, or specify a different location to modify it.

OutputAnaconda3 will now be installed into this location:
/home/sammy/anaconda3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/home/sammy/anaconda3] >>>

The installation process will continue. Note that it may take some time.

Once installation is complete, you’ll receive the following output:

Output...
installation finished.
Do you wish the installer to prepend the Anaconda3 install location
to PATH in your /home/sammy/.bashrc ? [yes|no]
[no] >>>

Type yes so that you can use the conda command. You’ll receive the following output next:

OutputAppending source /home/sammy/anaconda3/bin/activate to /home/sammy/.bashrc
A backup will be made to: /home/sammy/.bashrc-anaconda3.bak
...

Finally, you’ll receive the following prompt regarding whether or not you would like to download Visual Studio Code (or VSCode), a free and open-source editor for code developed by Microsoft that can run on Linux. You can learn more about the editor on the official Visual Studio Code website.

At this point, you can decide whether or not to download the editor now by typing yes or no.

Anaconda is partnered with Microsoft! Microsoft VSCode is a streamlined
code editor with support for development operations like debugging, task
running and version control.

To install Visual Studio Code, you will need:
  - Administrator Privileges
  - Internet connectivity

Visual Studio Code License: https://code.visualstudio.com/license

Do you wish to proceed with the installation of Microsoft VSCode? [yes|no]
>>>

In order to activate the installation, you should source the ~/.bashrc file:

source ~/.bashrc

Once you have done that, you can verify your install by making use of the conda command, for example with list:

conda list

You’ll receive output of all the packages you have available through the Anaconda installation:

Output# packages in environment at /home/sammy/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0            py36he11e457_0  
alabaster                 0.7.10           py36h306e16b_0  
anaconda                  5.2.0                    py36_3  
...

Now that Anaconda is installed, we can go on to setting up Anaconda environments.

Setting Up Anaconda Environments

Anaconda virtual environments allow you to keep projects organized by Python versions and packages needed. For each Anaconda environment you set up, you can specify which version of Python to use and can keep all of your related programming files together within that directory.

First, we can check to see which versions of Python are available for us to use:

conda search "^python$"

You’ll receive output with the different versions of Python that you can target, including both Python 3 and Python 2 versions. Since we are using the Anaconda with Python 3 in this tutorial, you will have access only to the Python 3 versions of packages.

Let’s create an environment using the most recent version of Python 3. We can achieve this by assigning version 3 to the python argument. We’ll call the environment my_env, but you’ll likely want to use a more descriptive name for your environment especially if you are using environments to access more than one version of Python.

conda create --name my_env python=3

We’ll receive output with information about what is downloaded and which packages will be installed, and then be prompted to proceed with y or n. As long as you agree, type y.

The conda utility will now fetch the packages for the environment and let you know when it’s complete.

You can activate your new environment by typing the following:

source activate my_env

With your environment activated, your command prompt prefix will change:

Within the environment, you can verify that you’re using the version of Python that you had intended to use:

 python --version

OutputPython 3.7.0 :: Anaconda, Inc.

When you’re ready to deactivate your Anaconda environment, you can do so by typing:

source deactivate

Note that you can replace the word source with . to achieve the same results.

To target a more specific version of Python, you can pass a specific version to the python argument, like 3.5, for example:

conda create -n my_env35 python=3.5

You can update your version of Python along the same branch (as in updating Python 3.5.1 to Python 3.5.2) within a respective environment with the following command:

conda update python

If you would like to target a more specific version of Python, you can pass that to the python argument, as in python=3.3.2.

You can inspect all of the environments you have set up with this command:

conda info --envs

Output# conda environments:
#
base                  *  /home/sammy/anaconda3
my_env                   /home/sammy/anaconda3/envs/my_env
my_env35                 /home/sammy/anaconda3/envs/my_env35

The asterisk indicates the current active environment.

Each environment you create with conda create will come with several default packages:

openssl
pip
python
readline
setuptools
sqlite
tk
wheel
xz
zlib

You can add additional packages, such as numpy for example, with the following command:

conda install --name my_env35 numpy

If you know you would like a numpy environment upon creation, you can target it in your conda create command:

conda create --name my_env python=3 numpy

If you are no longer working on a specific project and have no further need for the associated environment, you can remove it. To do so, type the following:

conda remove --name my_env35 --all

Now, when you type the conda info --envs command, the environment that you removed will no longer be listed.

Updating Anaconda

You should regularly ensure that Anaconda is up-to-date so that you are working with all the latest package releases.

To do this, you should first update the conda utility:

conda update conda

When prompted to do so, type y to proceed with the update.

Once the update of conda is complete, you can update the Anaconda distribution:

conda update anaconda

Again when prompted to do so, type y to proceed.

This will ensure that you are using the latest releases of conda and Anaconda.

Uninstalling Anaconda

If you are no longer using Anaconda and find that you need to uninstall it, you should start with the anaconda-clean module, which will remove configuration files for when you uninstall Anaconda.

conda install anaconda-clean

Type y when prompted to do so.

Once it is installed, you can run the following command. You will be prompted to answer y before deleting each one. If you would prefer not to be prompted, add --yes to the end of your command:

anaconda-clean

This will also create a backup folder called .anaconda_backup in your home directory:

Output
Backup directory: /home/sammy/.anaconda_backup/2018-09-06T183049

You can now remove your entire Anaconda directory by entering the following command:

rm -rf ~/anaconda3

Finally, you can remove the PATH line from your .bashrc file that Anaconda added. To do so, first open a text editor such as nano:

nano ~/.bashrc

Then scroll down to the end of the file (if this is a recent install) or type CTRL + W to search for Anaconda. Delete or comment out the export PATH line:

/home/sammy/.bashrc

...
# added by Anaconda3 installer
export PATH="/home/sammy/anaconda3/bin:$PATH"

When you’re done editing the file, type CTRL + X to exit and y to save changes.

Anaconda is now removed from your server.

Conclusion

This tutorial walked you through the installation of Anaconda, working with the conda command-line utility, setting up environments, updating Anaconda, and deleting Anaconda if you no longer need it.

You can use Anaconda to help you manage workloads for data science, scientific computing, analytics, and large-scale data processing. From here, you can check out our tutorials on data analysis and machine learning to learn more about various tools available to use and projects that you can do.

How To Set Up a Node.js Application for Production on Debian 9

2018-09-06T17:39:29Z

Introduction

Node.js is an open-source JavaScript runtime environment for building server-side and networking applications. The platform runs on Linux, macOS, FreeBSD, and Windows. Though you can run Node.js applications at the command line, this tutorial will focus on running them as a service. This means that the applications will restart on reboot or failure and are safe for use in a production environment.

In this tutorial, you will set up a production-ready Node.js environment on a single Debian 9 server. This server will run a Node.js application managed by PM2, and provide users with secure access to the application through an Nginx reverse proxy. The Nginx server will offer HTTPS, using a free certificate provided by Let's Encrypt.

Prerequisites

This guide assumes that you have the following:

A Debian 9 server setup, as described in the initial server setup guide for Debian 9. You should have a non-root user with sudo privileges and an active firewall.
A domain name pointed at your server's public IP. This tutorial will use the domain name example.com throughout.
Nginx installed, as covered in How To Install Nginx on Debian 9.
Nginx configured with SSL using Let's Encrypt certificates. How To Secure Nginx with Let's Encrypt on Debian 9 will walk you through the process.

When you've completed the prerequisites, you will have a server serving your domain's default placeholder page at https://example.com/.

Step 1 — Installing Node.js

Let's begin by installing the latest LTS release of Node.js, using the NodeSource package archives.

To install the NodeSource PPA and access its contents, you will first need to update your package index and install curl:

sudo apt update
sudo apt install curl

Make sure you're in your home directory, and then use curl to retrieve the installation script for the Node.js 8.x archives:

cd ~
curl -sL https://deb.nodesource.com/setup_8.x -o nodesource_setup.sh

You can inspect the contents of this script with nano or your preferred text editor:

nano nodesource_setup.sh

When you're done inspecting the script, run it under sudo:

sudo bash nodesource_setup.sh

The PPA will be added to your configuration and your local package cache will be updated automatically. After running the setup script from Nodesource, you can install the Node.js package:

sudo apt install nodejs

To check which version of Node.js you have installed after these initial steps, type:

nodejs -v

Output
v8.11.4

Note: When installing from the NodeSource PPA, the Node.js executable is called nodejs, rather than node.

The nodejs package contains the nodejs binary as well as npm, a package manager for Node modules, so you don't need to install npm separately.

npm uses a configuration file in your home directory to keep track of updates. It will be created the first time you run npm. Execute this command to verify that npm is installed and to create the configuration file:

npm -v

Output
5.6.0

In order for some npm packages to work (those that require compiling code from source, for example), you will need to install the build-essential package:

sudo apt install build-essential

You now have the necessary tools to work with npm packages that require compiling code from source.

With the Node.js runtime installed, let's move on to writing a Node.js application.

Step 2 — Creating a Node.js Application

Let's write a Hello World application that returns "Hello World" to any HTTP requests. This sample application will help you get Node.js set up. You can replace it with your own application — just make sure that you modify your application to listen on the appropriate IP addresses and ports.

First, let's create a sample application called hello.js:

cd ~
nano hello.js

Insert the following code into the file:

~/hello.js

const http = require('http');

const hostname = 'localhost';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World!\n');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Save the file and exit the editor.

This Node.js application listens on the specified address (localhost) and port (3000), and returns "Hello World!" with a 200 HTTP success code. Since we're listening on localhost, remote clients won't be able to connect to our application.

To test your application, type:

node hello.js

You will see the following output:

OutputServer running at http://localhost:3000/

Note: Running a Node.js application in this manner will block additional commands until the application is killed by pressing CTRL+C.

To test the application, open another terminal session on your server, and connect to localhost with curl:

curl http://localhost:3000

If you see the following output, the application is working properly and listening on the correct address and port:

OutputHello World!

If you do not see the expected output, make sure that your Node.js application is running and configured to listen on the proper address and port.

Once you're sure it's working, kill the application (if you haven't already) by pressing CTRL+C.

Step 3 — Installing PM2

Next let's install PM2, a process manager for Node.js applications. PM2 makes it possible to daemonize applications so that they will run in the background as a service.

Use npm to install the latest version of PM2 on your server:

sudo npm install pm2@latest -g

The -g option tells npm to install the module globally, so it's available system-wide.

Let's first use the pm2 start command to run your application, hello.js, in the background:

pm2 start hello.js

This also adds your application to PM2's process list, which is outputted every time you start an application:

Output[PM2] Spawning PM2 daemon with pm2_home=/home/sammy/.pm2
[PM2] PM2 Successfully daemonized
[PM2] Starting /home/sammy/hello.js in fork_mode (1 instance)
[PM2] Done.
┌──────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬───────┬──────────┐
│ App name │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ user  │ watching │
├──────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼───────┼──────────┤
│ hello    │ 0  │ fork │ 1338 │ online │ 0       │ 0s     │ 0%  │ 23.0 MB   │ sammy │ disabled │
└──────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴───────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

As you can see, PM2 automatically assigns an App name (based on the filename, without the .js extension) and a PM2 id. PM2 also maintains other information, such as the PID of the process, its current status, and memory usage.

Applications that are running under PM2 will be restarted automatically if the application crashes or is killed, but we can take an additional step to get the application to launch on system startup using the startup subcommand. This subcommand generates and configures a startup script to launch PM2 and its managed processes on server boots:

pm2 startup systemd

The last line of the resulting output will include a command to run with superuser privileges to set PM2 to start on boot:

Output[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sammy --hp /home/sammy

Run the command from the output, with your username in place of sammy:

sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sammy --hp /home/sammy

As an additional step, we can save the PM2 process list and corresponding environments:

pm2 save

You have now created a systemd unit that runs pm2 for your user on boot. This pm2 instance, in turn, runs hello.js.

Start the service with systemctl:

sudo systemctl start pm2-sammy

Check the status of the systemd unit:

systemctl status pm2-sammy

For a detailed overview of systemd, see Systemd Essentials: Working with Services, Units, and the Journal.

In addition to those we have covered, PM2 provides many subcommands that allow you to manage or look up information about your applications.

Stop an application with this command (specify the PM2 App name or id):

pm2 stop app_name_or_id

Restart an application:

pm2 restart app_name_or_id

List the applications currently managed by PM2:

pm2 list

Get information about a specific application using its App name:

pm2 info app_name

The PM2 process monitor can be pulled up with the monit subcommand. This displays the application status, CPU, and memory usage:

pm2 monit

Note that running pm2 without any arguments will also display a help page with example usage.

Now that your Node.js application is running and managed by PM2, let's set up the reverse proxy.

Step 4 — Setting Up Nginx as a Reverse Proxy Server

Your application is running and listening on localhost, but you need to set up a way for your users to access it. We will set up the Nginx web server as a reverse proxy for this purpose.

In the prerequisite tutorial, you set up your Nginx configuration in the /etc/nginx/sites-available/example.com file. Open this file for editing:

sudo nano /etc/nginx/sites-available/example.com

Within the server block, you should have an existing location / block. Replace the contents of that block with the following configuration. If your application is set to listen on a different port, update the highlighted portion to the correct port number:

/etc/nginx/sites-available/example.com

server {
...
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
...
}

This configures the server to respond to requests at its root. Assuming our server is available at example.com, accessing https://example.com/ via a web browser would send the request to hello.js, listening on port 3000 at localhost.

You can add additional location blocks to the same server block to provide access to other applications on the same server. For example, if you were also running another Node.js application on port 3001, you could add this location block to allow access to it via https://example.com/app2:

/etc/nginx/sites-available/example.com — Optional

server {
...
    location /app2 {
        proxy_pass http://localhost:3001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
...
}

Once you are done adding the location blocks for your applications, save the file and exit your editor.

Make sure you didn't introduce any syntax errors by typing:

sudo nginx -t

Restart Nginx:

sudo systemctl restart nginx

Assuming that your Node.js application is running, and your application and Nginx configurations are correct, you should now be able to access your application via the Nginx reverse proxy. Try it out by accessing your server's URL (its public IP address or domain name).

Conclusion

Congratulations! You now have your Node.js application running behind an Nginx reverse proxy on a Debian 9 server. This reverse proxy setup is flexible enough to provide your users access to other applications or static web content that you want to share.

How To Install Apache Tomcat 9 on Debian 9

2018-09-05T21:00:16Z

Introduction

Apache Tomcat is a web server and servlet container that is used to serve Java applications. Tomcat is an open source implementation of the Java Servlet and JavaServer Pages technologies, released by the Apache Software Foundation. This tutorial covers the basic installation and some configuration of the latest release of Tomcat 9 on your Debian 9 server.

Prerequisites

Before you begin with this guide, you should have a non-root user with sudo privileges set up on your server. You can learn how to do this by completing our Debian 9 initial server setup guide.

Step 1 — Install Java

Tomcat requires Java to be installed on the server so that any Java web application code can be executed. We can satisfy that requirement by installing OpenJDK with apt.

First, update your apt package index:

sudo apt update

Then install the Java Development Kit package with apt:

sudo apt install default-jdk

Now that Java is installed, we can create a tomcat user, which will be used to run the Tomcat service.

Step 2 — Create Tomcat User

For security purposes, Tomcat should be run as an unprivileged user (i.e. not root). We will create a new user and group that will run the Tomcat service.

Note: In some environments, a package called unscd may be installed by default in order to speed up requests to name servers like LDAP. The most recent version currently available in Debian contains a bug that causes certain commands (like the adduser command below) to produce additional output that looks like this:

sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting

These messages are harmless, but if you wish to avoid them, it is safe to remove the unscd package if you do not not plan on using systems like LDAP for user information:

apt remove unscd

First, create a new tomcat group:

sudo groupadd tomcat

Next, create a new tomcat user. We'll make this user a member of the tomcat group, with a home directory of /opt/tomcat (where we will install Tomcat), and with a shell of /bin/false (so nobody can log into the account):

sudo useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat

Now that our tomcat user is set up, let's download and install Tomcat.

Step 3 — Install Tomcat

The best way to install Tomcat 9 is to download the latest binary release then configure it manually.

Find the latest version of Tomcat 9 at the Tomcat 9 Downloads page. At the time of writing, the latest version is 9.0.11, but you should use a later stable version if it is available. Under the Binary Distributions section, then under the Core list, copy the link to the "tar.gz".

Next, change to the /tmp directory on your server. This is a good directory to download ephemeral items, like the Tomcat tarball, which we won't need after extracting the Tomcat contents:

cd /tmp

We’ll use the curl command-line tool to download the tarball. Install curl:

sudo apt install curl

Now, use curl to download the link that you copied from the Tomcat website:

curl -O http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.11/bin/apache-tomcat-9.0.11.tar.gz

We will install Tomcat to the /opt/tomcat directory. Create the directory, then extract the archive to it with these commands:

sudo mkdir /opt/tomcat
sudo tar xzvf apache-tomcat-9*tar.gz -C /opt/tomcat --strip-components=1

Next, we can set up the proper user permissions for our installation.

Step 4 — Update Permissions

The tomcat user that we set up needs to have access to the Tomcat installation. We'll set that up now.

Change to the directory where we unpacked the Tomcat installation:

cd /opt/tomcat

Give the tomcat group ownership over the entire installation directory:

sudo chgrp -R tomcat /opt/tomcat

Next, give the tomcat group read access to the conf directory and all of its contents, and execute access to the directory itself:

sudo chmod -R g+r conf
sudo chmod g+x conf

Make the tomcat user the owner of the webapps, work, temp, and logs directories:

sudo chown -R tomcat webapps/ work/ temp/ logs/

Now that the proper permissions are set up, we can create a systemd service file to manage the Tomcat process.

Step 5 — Create a systemd Service File

We want to be able to run Tomcat as a service, so we will set up systemd service file.

Tomcat needs to know where Java is installed. This path is commonly referred to as "JAVA_HOME". The easiest way to look up that location is by running this command:

sudo update-java-alternatives -l

Output
java-1.8.0-openjdk-amd64       1081       /usr/lib/jvm/java-1.8.0-openjdk-amd64

Your JAVA_HOME is the output from the last column (highlighted in red). Given the example above, the correct JAVA_HOME for this server would be:

JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk-amd64

Your JAVA_HOME may be different.

With this piece of information, we can create the systemd service file. Open a file called tomcat.service in the /etc/systemd/system directory by typing:

sudo nano /etc/systemd/system/tomcat.service

Paste the following contents into your service file. Modify the value of JAVA_HOME if necessary to match the value you found on your system. You may also want to modify the memory allocation settings that are specified in CATALINA_OPTS:

/etc/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking

Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

When you are finished, save and close the file.

Next, reload the systemd daemon so that it knows about our service file:

sudo systemctl daemon-reload

Start the Tomcat service by typing:

sudo systemctl start tomcat

Double check that it started without errors by typing:

sudo systemctl status tomcat

You should see output similar to the following:

Output● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/etc/systemd/system/tomcat.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 20:47:44 UTC; 3s ago
  Process: 9037 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
 Main PID: 9046 (java)
    Tasks: 46 (limit: 4915)
   CGroup: /system.slice/tomcat.service
           └─9046 /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Dja

Sep 05 20:47:44 tomcat systemd[1]: Starting Apache Tomcat Web Application Container...
Sep 05 20:47:44 tomcat systemd[1]: Started Apache Tomcat Web Application Container.

This confirms that Tomcat is up and running on your server.

Step 6 — Adjust the Firewall and Test the Tomcat Server

Now that the Tomcat service is started, we can test to make sure the default page is available.

Before we do that, we need to adjust the firewall to allow our requests to get to the service. If you followed the prerequisites, you will have a ufw firewall enabled currently.

Tomcat uses port 8080 to accept conventional requests. Allow traffic to that port by typing:

sudo ufw allow 8080

With the firewall modified, you can access the default splash page by going to your domain or IP address followed by :8080 in a web browser:

Open in web browser
http://server_domain_or_IP:8080

You will see the default Tomcat splash page, in addition to other information. However, if you click the links for the Manager App, for instance, you will be denied access. We can configure that access next.

If you were able to successfully accessed Tomcat, now is a good time to enable the service file so that Tomcat automatically starts at boot:

sudo systemctl enable tomcat

Step 7 — Configure Tomcat Web Management Interface

In order to use the manager web app that comes with Tomcat, we must add a login to our Tomcat server. We will do this by editing the tomcat-users.xml file:

sudo nano /opt/tomcat/conf/tomcat-users.xml

You will want to add a user who can access the manager-gui and admin-gui (web apps that come with Tomcat). You can do so by defining a user, similar to the example below, between the tomcat-users tags. Be sure to change the username and password to something secure:

tomcat-users.xml — Admin User

<tomcat-users . . .>
    <user username="admin" password="password" roles="manager-gui,admin-gui"/>
</tomcat-users>

Save and close the file when you are finished.

By default, newer versions of Tomcat restrict access to the Manager and Host Manager apps to connections coming from the server itself. Since we are installing on a remote machine, you will probably want to remove or alter this restriction. To change the IP address restrictions on these, open the appropriate context.xml files.

For the Manager app, type:

sudo nano /opt/tomcat/webapps/manager/META-INF/context.xml

For the Host Manager app, type:

sudo nano /opt/tomcat/webapps/host-manager/META-INF/context.xml

Inside, comment out the IP address restriction to allow connections from anywhere. Alternatively, if you would like to allow access only to connections coming from your own IP address, you can add your public IP address to the list:

context.xml files for Tomcat webapps

<Context antiResourceLocking="false" privileged="true" >
  <!--<Valve className="org.apache.catalina.valves.RemoteAddrValve"
         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />-->
</Context>

Save and close the files when you are finished.

To put our changes into effect, restart the Tomcat service:

sudo systemctl restart tomcat

Step 8 — Access the Web Interface

Now that we have create a user, we can access the web management interface again in a web browser. Once again, you can get to the correct interface by entering your server's domain name or IP address followed on port 8080 in your browser:

Open in web browser
http://server_domain_or_IP:8080

The page you see should be the same one you were given when you tested earlier:

Let's take a look at the Manager App, accessible via the link or http://server_domain_or_IP:8080/manager/html. You will need to enter the account credentials that you added to the tomcat-users.xml file. Afterwards, you should see a page that looks like this:

The Web Application Manager is used to manage your Java applications. You can Start, Stop, Reload, Deploy, and Undeploy here. You can also run some diagnostics on your apps (i.e. find memory leaks). Lastly, information about your server is available at the very bottom of this page.

Now let's take a look at the Host Manager, accessible via the link or http://server_domain_or_IP:8080/host-manager/html/:

From the Virtual Host Manager page, you can add virtual hosts to serve your applications from.

Conclusion

Your installation of Tomcat is complete! Your are now free to deploy your own Java web applications!

Currently, your Tomcat installation is functional, but entirely unencrypted. This means that all data, including sensitive items like passwords, are sent in plain text that can be intercepted and read by other parties on the internet. In order to prevent this from happening, it is strongly recommended that you encrypt your connections with SSL. You can find out how to encrypt your connections to Tomcat by following this guide (note: this guide covers Tomcat 8 encryption on Ubuntu 16.04).

How To Install and Secure phpMyAdmin on Debian 9

2018-09-05T21:01:44Z

Introduction

While many users need the functionality of a database management system like MariaDB, they may not feel comfortable interacting with the system solely from the MariaDB prompt.

phpMyAdmin was created so that users can interact with MariaDB through a web interface. In this guide, we'll discuss how to install and secure phpMyAdmin so that you can safely use it to manage your databases on a Debian 9 system.

Prerequisites

Before you get started with this guide, you need to have some basic steps completed.

First, we'll assume that your server has a non-root user with sudo privileges, as well as a firewall configured with ufw, as described in the initial server setup guide for Debian 9.

We're also going to assume that you've completed a LAMP (Linux, Apache, MariaDB, and PHP) installation on your Debian 9 server. If you’ve not yet done this, follow our guide on installing a LAMP stack on Debian 9 to set this up.

Finally, there are important security considerations when using software like phpMyAdmin, since it:

Communicates directly with your MariaDB installation
Handles authentication using MariaDB credentials
Executes and returns results for arbitrary SQL queries

For these reasons, and because it is a widely-deployed PHP application which is frequently targeted for attack, you should never run phpMyAdmin on remote systems over a plain HTTP connection. If you do not have an existing domain configured with an SSL/TLS certificate, you can follow this guide on securing Apache with Let's Encrypt on Debian 9. This will require you to register a domain name, create DNS records for your server, and set up an Apache Virtual Host.

Once you are finished with these steps, you're ready to get started with this guide.

Step 1 — Installing phpMyAdmin

To get started, we will install phpMyAdmin from the default Debian repositories.

This is done by updating your server’s package index and then using the apt packaging system to pull down the files and install them on your system:

sudo apt update
sudo apt install phpmyadmin php-mbstring php-gettext

This will ask you a few questions in order to configure your installation correctly.

Warning: When the prompt appears, “apache2” is highlighted, but not selected. If you do not hit SPACE to select Apache, the installer will not move the necessary files during installation. Hit SPACE, TAB, and then ENTER to select Apache.

For the server selection, choose apache2
Select Yes when asked whether to use dbconfig-common to set up the database
You will then be asked to choose and confirm a MySQL application password for phpMyAdmin

Note: MariaDB is a community-developed fork of MySQL, and although the two programs are closely related, they are not completely interchangeable. While phpMyAdmin was designed specifically for managing MySQL databases and makes reference to MySQL in various dialogue boxes, rest assured that your installation of MariaDB will work correctly with phpMyAdmin.

The installation process adds the phpMyAdmin Apache configuration file into the /etc/apache2/conf-enabled/ directory, where it is read automatically. The only thing you need to do is explicitly enable the mbstring PHP extension which is used to manage non-ASCII strings and convert strings to different encodings. Do this by typing:

sudo phpenmod mbstring

Afterwards, restart Apache for your changes to be recognized:

sudo systemctl restart apache2

phpMyAdmin is now installed and configured. However, before you can log in and begin managing your MariaDB databases, you will need to ensure that your MariaDB users have the privileges required for interacting with the program.

Step 2 — Adjusting User Authentication and Privileges

When you installed phpMyAdmin onto your server, it automatically created a database user called phpmyadmin which performs certain underlying processes for the program. Rather than logging in as this user with the administrative password you set during installation, it’s recommended that you log in using a different account.

In new installs on Debian systems, the root MariaDB user is set to authenticate using the unix_socket plugin by default rather than with a password. This allows for some greater security and usability in many cases, but it can also complicate things when you need to allow an external program (e.g., phpMyAdmin) administrative rights through this user. Because the server uses the root account for tasks like log rotation and starting and stopping the server, it is best not to change the root account's authentication details. Since phpMyAdmin requires users to authenticate with a password, you will need to create a new MariaDB account in order to access the interface.

If you followed the prerequisite tutorial on installing a LAMP stack and created a MariaDB user account as described in Step 2, you can just log in to phpMyAdmin under that account using the password you created when you set it up by visiting this link:

https://your_domain_or_IP/phpmyadmin

If you haven’t created a MariaDB user, or if you have but you’d like to create another user just for the purpose of managing databases through phpMyAdmin, continue on with this section to learn how to set one up.

Begin by opening up the MariaDB shell:

sudo mariadb

Note: If you have password authentication enabled, as you would if you’ve already created a new user account for your MariaDB server, you will need to use a different command to access the MariaDB shell. The following will run your MariaDB client with regular user privileges, and you will only gain administrator privileges within the database by authenticating:

mariadb -u user -p

From there, create a new user and give it a strong password:

CREATE USER 'sammy'@'localhost' IDENTIFIED BY 'password';

Then, grant your new user appropriate privileges. For example, you could grant the user privileges to all tables within the database, as well as the power to add, change, and remove user privileges, with this command:

GRANT ALL PRIVILEGES ON *.* TO 'sammy'@'localhost' WITH GRANT OPTION;

Following that, exit the MariaDB shell:

exit

You can now access the web interface by visiting your server's domain name or public IP address, followed by /phpmyadmin:

https://your_domain_or_IP/phpmyadmin

When you log in, you'll see the user interface, which will look something like this:

Now that you’re able to connect and interact with phpMyAdmin, all that’s left to do is harden your system’s security to protect it from attackers.

Step 3 — Securing Your phpMyAdmin Instance

Because of its ubiquity, phpMyAdmin is a popular target for attackers, and you should take extra care to prevent unauthorized access. One of the easiest ways of doing this is to place a gateway in front of the entire application by using Apache's built-in .htaccess authentication and authorization functionalities.

To do this, you must first enable the use of .htaccess file overrides by editing your Apache configuration file.

Edit the linked file that has been placed in your Apache configuration directory:

sudo nano /etc/apache2/conf-available/phpmyadmin.conf

Add an AllowOverride All directive within the <Directory /usr/share/phpmyadmin> section of the configuration file, like this:

/etc/apache2/conf-available/phpmyadmin.conf

<Directory /usr/share/phpmyadmin>
    Options FollowSymLinks
    DirectoryIndex index.php
    AllowOverride All
    . . .

When you have added this line, save and close the file.

To implement the changes you made, restart Apache:

sudo systemctl restart apache2

Now that you have enabled .htaccess use for your application, you need to create one to actually implement some security.

In order for this to be successful, the file must be created within the application directory. You can create the necessary file and open it in your text editor with root privileges by typing:

sudo nano /usr/share/phpmyadmin/.htaccess

Within this file, enter the following information:

/usr/share/phpmyadmin/.htaccess

AuthType Basic
AuthName "Restricted Files"
AuthUserFile /etc/phpmyadmin/.htpasswd
Require valid-user

Here is what each of these lines mean:

AuthType Basic: This line specifies the authentication type that you are implementing. This type will implement password authentication using a password file.
AuthName: This sets the message for the authentication dialog box. You should keep this generic so that unauthorized users won't gain any information about what is being protected.
AuthUserFile: This sets the location of the password file that will be used for authentication. This should be outside of the directories that are being served. We will create this file shortly.
Require valid-user: This specifies that only authenticated users should be given access to this resource. This is what actually stops unauthorized users from entering.

When you are finished, save and close the file.

The location that you selected for your password file was /etc/phpmyadmin/.htpasswd. You can now create this file and pass it an initial user with the htpasswd utility:

sudo htpasswd -c /etc/phpmyadmin/.htpasswd username

You will be prompted to select and confirm a password for the user you are creating. Afterwards, the file is created with the hashed password that you entered.

If you want to enter an additional user, you need to do so without the -c flag, like this:

sudo htpasswd /etc/phpmyadmin/.htpasswd additionaluser

Now, when you access your phpMyAdmin subdirectory, you will be prompted for the additional account name and password that you just configured:

https://domain_name_or_IP/phpmyadmin

After entering the Apache authentication, you'll be taken to the regular phpMyAdmin authentication page to enter your MariaDB credentials. This setup adds an additional layer of security, which is desirable since phpMyAdmin has suffered from vulnerabilities in the past.

Conclusion

You should now have phpMyAdmin configured and ready to use on your Debian 9 server. Using this interface, you can easily create databases, users, tables, etc., and perform the usual operations like deleting and modifying structures and data.

How To Secure Apache with Let's Encrypt on Debian 9

2018-09-05T20:42:04Z

Introduction

Let's Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free TLS/SSL certificates, thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx.

In this tutorial, you will use Certbot to obtain a free SSL certificate for Apache on Debian 9 and set up your certificate to renew automatically.

This tutorial will use a separate Apache virtual host file instead of the default configuration file. We recommend creating new Apache virtual host files for each domain because it helps to avoid common mistakes and maintains the default files as a fallback configuration.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server set up by following this initial server setup for Debian 9 tutorial, including a non-root user with sudo privileges and a firewall.
A fully registered domain name. This tutorial will use example.com throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.
Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.
- An A record with example.com pointing to your server's public IP address.
- An A record with www.example.com pointing to your server's public IP address.
Apache installed by following How To Install Apache on Debian 9. Be sure that you have a virtual host file for your domain. This tutorial will use /etc/apache2/sites-available/example.com.conf as an example.

Step 1 — Installing Certbot

The first step to using Let's Encrypt to obtain an SSL certificate is to install the Certbot software on your server.

As of this writing, Certbot is not available from the Debian software repositories by default. In order to download the software using apt, you will need to add the backports repository to your sources.list file where apt looks for package sources. Backports are packages from Debian’s testing and unstable distributions that are recompiled so they will run without new libraries on stable Debian distributions.

To add the backports repository, open (or create) the sources.list file in your /etc/apt/ directory:

sudo nano /etc/apt/sources.list

At the bottom of the file, add the following line:

/etc/apt/sources.list.d/sources.list

. . .
deb http://ftp.debian.org/debian stretch-backports main

This includes the main packages, which are Debian Free Software Guidelines (DFSG)-compliant, as well as the non-free and contrib components, which are either not DFSG-compliant themselves or include dependencies in this category.

Save and close the file by pressing CTRL+X, Y, then ENTER, then update your package lists:

sudo apt update

Then install Certbot with the following command. Note that the -t option tells apt to search for the package by looking in the backports repository you just added:

sudo apt install python-certbot-apache -t stretch-backports

Certbot is now ready to use, but in order for it to configure SSL for Apache, we need to verify that Apache has been configured correctly.

Step 2 — Setting Up the SSL Certificate

Certbot needs to be able to find the correct virtual host in your Apache configuration for it to automatically configure SSL. Specifically, it does this by looking for a ServerName directive that matches the domain you request a certificate for.

If you followed the virtual host set up step in the Apache installation tutorial, you should have a VirtualHost block for your domain at /etc/apache2/sites-available/example.com.conf with the ServerName directive already set appropriately.

To check, open the virtual host file for your domain using nano or your favorite text editor:

sudo nano /etc/apache2/sites-available/example.com.conf

Find the existing ServerName line. It should look like this, with your own domain name instead of example.com:

/etc/apache2/sites-available/example.com.conf

...
ServerName example.com;
...

If it doesn’t already, update the ServerName directive to point to your domain name. Then save the file, quit your editor, and verify the syntax of your configuration edits:

sudo apache2ctl configtest

If there aren't any syntax errors, you will see this output:

OutputSyntax OK

If you get an error, reopen the virtual host file and check for any typos or missing characters. Once your configuration file's syntax is correct, reload Apache to load the new configuration:

sudo systemctl reload apache2

Certbot can now find the correct VirtualHost block and update it.

Next, let's update the firewall to allow HTTPS traffic.

Step 3 — Allowing HTTPS Through the Firewall

If you have the ufw firewall enabled, as recommended by the prerequisite guides, you'll need to adjust the settings to allow for HTTPS traffic. Luckily, when installed on Debian, ufw comes packaged with a few profiles that help to simplify the process of changing firewall rules for HTTP and HTTPS traffic.

You can see the current setting by typing:

sudo ufw status

If you followed the Step 2 of our guide on How to Install Apache on Debian 9, the output of this command will look like this, showing that only HTTP traffic is allowed to the web server:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
WWW                        ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
WWW (v6)                   ALLOW       Anywhere (v6)

To additionally let in HTTPS traffic, allow the “WWW Full” profile and delete the redundant “WWW” profile allowance:

sudo ufw allow 'WWW Full'
sudo ufw delete allow 'WWW'

Your status should now look like this:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
WWW Full                   ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
WWW Full (v6)              ALLOW       Anywhere (v6)

Next, let's run Certbot and fetch our certificates.

Step 4 — Obtaining an SSL Certificate

Certbot provides a variety of ways to obtain SSL certificates through plugins. The Apache plugin will take care of reconfiguring Apache and reloading the config whenever necessary. To use this plugin, type the following:

sudo certbot --apache -d example.com -d www.example.com

This runs certbot with the --apache plugin, using -d to specify the names you'd like the certificate to be valid for.

If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let's Encrypt server, then run a challenge to verify that you control the domain you're requesting a certificate for.

If that's successful, certbot will ask how you'd like to configure your HTTPS settings:

OutputPlease choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Select your choice then hit ENTER. The configuration will be updated, and Apache will reload to pick up the new settings. certbot will wrap up with a message telling you the process was successful and where your certificates are stored:

OutputIMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2018-12-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Your certificates are downloaded, installed, and loaded. Try reloading your website using https:// and notice your browser's security indicator. It should indicate that the site is properly secured, usually with a green lock icon. If you test your server using the SSL Labs Server Test, it will get an A grade.

Let's finish by testing the renewal process.

Step 5 — Verifying Certbot Auto-Renewal

Let's Encrypt's certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process. The certbot package we installed takes care of this for us by adding a renew script to /etc/cron.d. This script runs twice a day and will automatically renew any certificate that's within thirty days of expiration.

To test the renewal process, you can do a dry run with certbot:

sudo certbot renew --dry-run

If you see no errors, you're all set. When necessary, Certbot will renew your certificates and reload Apache to pick up the changes. If the automated renewal process ever fails, Let’s Encrypt will send a message to the email you specified, warning you when your certificate is about to expire.

Conclusion

In this tutorial, you installed the Let's Encrypt client certbot, downloaded SSL certificates for your domain, configured Apache to use these certificates, and set up automatic certificate renewal. If you have further questions about using Certbot, their documentation is a good place to start.

How To Install and Secure Redis on Debian 9

2018-09-05T20:33:58Z

Introduction

Redis is an in-memory key-value store known for its flexibility, performance, and wide language support. This tutorial demonstrates how to install, configure, and secure Redis on a Debian 9 server.

Prerequisites

To complete this guide, you will need access to a Debian 9 server that has a non-root user with sudo privileges and a basic firewall configured. You can set this up by following our Initial Server Setup guide.

When you are ready to begin, log in to your server as your sudo-enabled user and continue below.

Step 1 — Installing and Configuring Redis

In order to get the latest version of Redis, we will use apt to install it from the official Debian repositories.

Update your local apt package cache and install Redis by typing:

sudo apt update
sudo apt install redis-server

This will download and install Redis and its dependencies. Following this, there is one important configuration change to make in the Redis configuration file, which was generated automatically during the installation.

Open this file with your preferred text editor:

sudo nano /etc/redis/redis.conf

Inside the file, find the supervised directive. This directive allows you to declare an init system to manage Redis as a service, providing you with more control over its operation. The supervised directive is set to no by default. Since you are running Debian, which uses the systemd init system, change this to systemd:

/etc/redis/redis.conf

. . .

# If you run Redis from upstart or systemd, Redis can interact with your
# supervision tree. Options:
#   supervised no      - no supervision interaction
#   supervised upstart - signal upstart by putting Redis into SIGSTOP mode
#   supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET
#   supervised auto    - detect upstart or systemd method based on
#                        UPSTART_JOB or NOTIFY_SOCKET environment variables
# Note: these supervision methods only signal "process is ready."
#       They do not enable continuous liveness pings back to your supervisor.
supervised systemd

. . .

That’s the only change you need to make to the Redis configuration file at this point, so save and close it when you are finished. Then, reload the Redis service file to reflect the changes you made to the configuration file:

sudo systemctl restart redis

With that, you’ve installed and configured Redis and it’s running on your machine. Before you begin using it, though, it’s prudent to first check whether Redis is functioning correctly.

Step 2 — Testing Redis

As with any newly-installed software, it’s a good idea to ensure that Redis is functioning as expected before making any further changes to its configuration. We will go over a handful of ways to check that Redis is working correctly in this step.

Start by checking that the Redis service is running:

sudo systemctl status redis

If it is running without any errors, this command will produce output similar to the following:

Output● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 20:19:44 UTC; 41s ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 10829 ExecStopPost=/bin/run-parts --verbose /etc/redis/redis-server.post-down.d (code=exited, status=0/SUCCESS)
  Process: 10825 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
  Process: 10823 ExecStop=/bin/run-parts --verbose /etc/redis/redis-server.pre-down.d (code=exited, status=0/SUCCESS)
  Process: 10842 ExecStartPost=/bin/run-parts --verbose /etc/redis/redis-server.post-up.d (code=exited, status=0/SUCCESS)
  Process: 10838 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
  Process: 10834 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d (code=exited, status=0/SUCCESS)
 Main PID: 10841 (redis-server)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/redis-server.service
           └─10841 /usr/bin/redis-server 127.0.0.1:6379
. . .

Here, you can see that Redis is running and is already enabled, meaning that it is set to start up every time the server boots.

Note: This setting is desirable for many common use cases of Redis. If, however, you prefer to start up Redis manually every time your server boots, you can configure this with the following command:

sudo systemctl disable redis

To test that Redis is functioning correctly, connect to the server using the command-line client:

redis-cli

In the prompt that follows, test connectivity with the ping command:

ping

OutputPONG

This output confirms that the server connection is still alive. Next, check that you’re able to set keys by running:

set test "It's working!"

OutputOK

Retrieve the value by typing:

get test

Assuming everything is working, you will be able to retrieve the value you stored:

Output"It's working!"

After confirming that you can fetch the value, exit the Redis prompt to get back to the shell:

exit

As a final test, we will check whether Redis is able to persist data even after it’s been stopped or restarted. To do this, first restart the Redis instance:

sudo systemctl restart redis

Then connect with the command-line client once again and confirm that your test value is still available:

redis-cli

get test

The value of your key should still be accessible:

Output"It's working!"

Exit out into the shell again when you are finished:

exit

With that, your Redis installation is fully operational and ready for you to use. However, some of its default configuration settings are insecure and provide malicious actors with opportunities to attack and gain access to your server and its data. The remaining steps in this tutorial cover methods for mitigating these vulnerabilities, as prescribed by the official Redis website. Although these steps are optional and Redis will still function if you choose not to follow them, it is strongly recommended that you complete them in order to harden your system’s security.

Step 3 — Binding to localhost

By default, Redis is only accessible from localhost. However, if you installed and configured Redis by following a different tutorial than this one, you might have updated the configuration file to allow connections from anywhere. This is not as secure as binding to localhost.

To correct this, open the Redis configuration file for editing:

sudo nano /etc/redis/redis.conf

Locate this line and make sure it is uncommented (remove the # if it exists):

/etc/redis/redis.conf

bind 127.0.0.1

Save and close the file when finished (press CTRL + X, Y, then ENTER).

Then, restart the service to ensure that systemd reads your changes:

sudo systemctl restart redis

To check that this change has gone into effect, run the following netstat command:

sudo netstat -lnp | grep redis

Outputtcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      10959/redis-server

This output shows that the redis-server program is bound to localhost (127.0.0.1), reflecting the change you just made to the configuration file. If you see another IP address in that column (0.0.0.0, for example), then you should double check that you uncommented the correct line and restart the Redis service again.

Now that your Redis installation is only listening in on localhost, it will be more difficult for malicious actors to make requests or gain access to your server. However, Redis isn’t currently set to require users to authenticate themselves before making changes to its configuration or the data it holds. To remedy this, Redis allows you to require users to authenticate with a password before making changes via the Redis client (redis-cli).

Step 4 — Configuring a Redis Password

Configuring a Redis password enables one of its two built-in security features — the auth command, which requires clients to authenticate to access the database. The password is configured directly in Redis's configuration file, /etc/redis/redis.conf, so open that file again with your preferred editor:

sudo nano /etc/redis/redis.conf

Scroll to the SECURITY section and look for a commented directive that reads:

/etc/redis/redis.conf

# requirepass foobared

Uncomment it by removing the #, and change foobared to a secure password.

Note: Above the requirepass directive in the redis.conf file, there is a commented warning:

# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#

Thus, it’s important that you specify a very strong and very long value as your password. Rather than make up a password yourself, you can use the openssl command to generate a random one, as in the following example. By piping the output of the first command to the second openssl command, as shown here, it will remove any line breaks produced by that the first command:

openssl rand 60 | openssl base64 -A

Your output should look something like:

OutputRBOJ9cCNoGCKhlEBwQLHri1g+atWgn4Xn4HwNUbtzoVxAYxkiYBi7aufl4MILv1nxBqR4L6NNzI0X6cE

After copying and pasting the output of that command as the new value for requirepass, it should read:

/etc/redis/redis.conf
requirepass RBOJ9cCNoGCKhlEBwQLHri1g+atWgn4Xn4HwNUbtzoVxAYxkiYBi7aufl4MILv1nxBqR4L6NNzI0X6cE

After setting the password, save and close the file, then restart Redis:

sudo systemctl restart redis.service

To test that the password works, access the Redis command line:

redis-cli

The following shows a sequence of commands used to test whether the Redis password works. The first command tries to set a key to a value before authentication:

set key1 10

That won't work because you didn’t authenticate, so Redis returns an error:

Output(error) NOAUTH Authentication required.

The next command authenticates with the password specified in the Redis configuration file:

auth your_redis_password

Redis acknowledges:

OutputOK

After that, running the previous command again will succeed:

set key1 10

OutputOK

get key1 queries Redis for the value of the new key.

get key1

Output"10"

After confirming that you’re able to run commands in the Redis client after authenticating, you can exit the redis-cli:

quit

Next, we'll look at renaming Redis commands which, if entered by mistake or by a malicious actor, could cause serious damage to your machine.

Step 5 — Renaming Dangerous Commands

The other security feature built into Redis involves renaming or completely disabling certain commands that are considered dangerous.

When run by unauthorized users, such commands can be used to reconfigure, destroy, or otherwise wipe your data. Like the authentication password, renaming or disabling commands is configured in the same SECURITY section of the /etc/redis/redis.conf file.

Some of the commands that are considered dangerous include: FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, and DEBUG. This is not a comprehensive list, but renaming or disabling all of the commands in that list is a good starting point for enhancing your Redis server’s security.

Whether you should disable or rename a command depends on your specific needs or those of your site. If you know you will never use a command that could be abused, then you may disable it. Otherwise, it might be in your best interest to rename it.

To enable or disable Redis commands, open the configuration file once more:

sudo nano  /etc/redis/redis.conf

Warning: The following steps showing how to disable and rename commands are examples. You should only choose to disable or rename the commands that make sense for you. You can review the full list of commands for yourself and determine how they might be misused at redis.io/commands.

To disable a command, simply rename it to an empty string (signified by a pair of quotation marks with no characters between them), as shown below:

/etc/redis/redis.conf

. . .
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command DEBUG ""
. . .

To rename a command, give it another name as shown in the examples below. Renamed commands should be difficult for others to guess, but easy for you to remember:

/etc/redis/redis.conf

. . .
# rename-command CONFIG ""
rename-command SHUTDOWN SHUTDOWN_MENOT
rename-command CONFIG ASC12_CONFIG
. . .

Save your changes and close the file.

After renaming a command, apply the change by restarting Redis:

sudo systemctl restart redis

To test the new command, enter the Redis command line:

redis-cli

Then, authenticate:

auth your_redis_password

OutputOK

Let’s assume that you renamed the CONFIG command to ASC12_CONFIG, as in the preceding example. First, try using the original CONFIG command. It should fail, because you’ve renamed it:

config get requirepass

Output(error) ERR unknown command 'config'

Calling the renamed command, however, will be successful. It is not case-sensitive:

asc12_config get requirepass

Output1) "requirepass"
2) "your_redis_password"

Finally, you can exit from redis-cli:

exit

Note that if you're already using the Redis command line and then restart Redis, you'll need to re-authenticate. Otherwise, you'll get this error if you type a command:

OutputNOAUTH Authentication required.

Regarding the practice of renaming commands, there's a cautionary statement at the end of the SECURITY section in /etc/redis/redis.conf which reads:

Please note that changing the name of commands that are logged into the AOF file or transmitted to slaves may cause problems.

Note: The Redis project chooses to use the terms “master” and “slave” while DigitalOcean generally prefers alternative descriptors. In order to avoid confusion we’ve chosen to use the terms used in the Redis documentation here.

That means if the renamed command is not in the AOF file, or if it is but the AOF file has not been transmitted to slaves, then there should be no problem.

So, keep that in mind when you're trying to rename commands. The best time to rename a command is when you're not using AOF persistence, or right after installation, that is, before your Redis-using application has been deployed.

When you're using AOF and dealing with a master-slave installation, consider this answer from the project's GitHub issue page. The following is a reply to the author's question:

The commands are logged to the AOF and replicated to the slave the same way they are sent, so if you try to replay the AOF on an instance that doesn't have the same renaming, you may face inconsistencies as the command cannot be executed (same for slaves).

Thus, the best way to handle renaming in cases like that is to make sure that renamed commands are applied to all instances in master-slave installations.

Conclusion

In this tutorial, you installed and configured Redis, validated that your Redis installation is functioning correctly, and used its built-in security features to make it less vulnerable to attacks from malicious actors.

Keep in mind that once someone is logged in to your server, it's very easy to circumvent the Redis-specific security features we've put in place. Therefore, the most important security feature on your Redis server is your firewall (which you configured if you followed the prerequisite Initial Server Setup tutorial), as this makes it extremely difficult for malicious actors to jump that fence.

How To Create RAID Arrays with mdadm on Debian 9

2018-09-05T20:20:35Z

Introduction

The mdadm utility can be used to create and manage storage arrays using Linux's software RAID capabilities. Administrators have great flexibility in coordinating their individual storage devices and creating logical storage devices that have greater performance or redundancy characteristics.

In this guide, we will go over a number of different RAID configurations that can be set up using a Debian 9 server.

Prerequisites

In order to complete the steps in this guide, you should have:

A non-root user with sudo privileges on a Debian 9 server: The steps in this guide will be completed with a sudo user. To learn how to set up an account with these privileges, follow our Debian 9 initial server setup guide.
A basic understanding of RAID terminology and concepts: While this guide will touch on some RAID terminology in passing, a more complete understanding is very useful. To learn more about RAID and to get a better understanding of what RAID level is right for you, read our introduction to RAID article.
Multiple raw storage devices available on your server: We will be demonstrating how to configure various types of arrays on the server. As such, you will need some drives to configure. If you are using DigitalOcean, you can use Block Storage volumes to fill this role. Depending on the array type, you will need at minimum between two to four storage devices. These drives do not need to be formatted prior to following this guide.

Installing the RAID Administration Tools

Before we begin, we need to install mdadm, the tool that allows us to set up and manage software RAID arrays in Linux. This is available in Debian's default repositories.

Update the local package cache to retrieve an up-to-date list of available packages and then download and install the package:

sudo apt update
sudo apt install mdadm

This will install mdadm and all of its dependencies. Verify that the utility is installed by typing:

sudo mdadm -V

Outputmdadm - v3.4 - 28th January 2016

The application version should be displayed, indicating that mdadm is installed and ready to use.

Resetting Existing RAID Devices

Throughout this guide, we will be introducing the steps to create a number of different RAID levels. If you wish to follow along, you will likely want to reuse your storage devices after each section. This section can be referenced to learn how to quickly reset your component storage devices prior to testing a new RAID level. Skip this section for now if you have not yet set up any arrays.

Warning: This process will completely destroy the array and any data written to it. Make sure that you are operating on the correct array and that you have copied off any data you need to retain prior to destroying the array.

Find the active arrays in the /proc/mdstat file by typing:

cat /proc/mdstat

OutputPersonalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid0 sdc[1] sdd[0]
      209584128 blocks super 1.2 512k chunks

            unused devices: <none>

Unmount the array from the filesystem:

sudo umount /dev/md0

Then, stop and remove the array by typing:

sudo mdadm --stop /dev/md0

Find the devices that were used to build the array with the following command:

Warning: Keep in mind that the /dev/sd* names can change any time you reboot! Check them every time to make sure you are operating on the correct devices.

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE            TYPE MOUNTPOINT
sda      100G                   disk 
sdb      100G                   disk 
sdc      100G linux_raid_member disk 
sdd      100G linux_raid_member disk 
vda       25G                   disk 
├─vda1  24.9G ext4              part /
├─vda14    4M                   part 
└─vda15  106M vfat              part /boot/efi

After discovering the devices used to create an array, zero their superblock to remove the RAID metadata and reset them to normal:

sudo mdadm --zero-superblock /dev/sdc
sudo mdadm --zero-superblock /dev/sdd

You should remove any of the persistent references to the array. Edit the /etc/fstab file and comment out or remove the reference to your array:

sudo nano /etc/fstab

/etc/fstab

. . .
# /dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0

Also, comment out or remove the array definition from the /etc/mdadm/mdadm.conf file:

sudo nano /etc/mdadm/mdadm.conf

/etc/mdadm/mdadm.conf

. . .
# ARRAY /dev/md0 metadata=1.2 name=mdadmwrite:0 UUID=7261fb9c:976d0d97:30bc63ce:85e76e91

Finally, update the initramfs again so that the early boot process does not try to bring an unavailable array online.

sudo update-initramfs -u

At this point, you should be ready to reuse the storage devices individually, or as components of a different array.

Creating a RAID 0 Array

The RAID 0 array works by breaking up data into chunks and striping it across the available disks. This means that each disk contains a portion of the data and that multiple disks will be referenced when retrieving information.

Requirements: minimum of 2 storage devices
Primary benefit: Performance
Things to keep in mind: Make sure that you have functional backups. A single device failure will destroy all data in the array.

Identifying the Component Devices

To get started, find the identifiers for the raw disks that you will be using:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE TYPE MOUNTPOINT
sda      100G        disk
sdb      100G        disk
vda       25G        disk 
├─vda1  24.9G ext4   part /
├─vda14    4M        part 
└─vda15  106M vfat   part /boot/efi

As you can see above, we have two disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda and /dev/sdb identifiers for this session. These will be the raw components we will use to build the array.

Creating the Array

To create a RAID 0 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sda /dev/sdb

You can ensure that the RAID was successfully created by checking the /proc/mdstat file:

cat /proc/mdstat

OutputPersonalities : [raid0] 
md0 : active raid0 sdb[1] sda[0]
      209584128 blocks super 1.2 512k chunks

unused devices: <none>

As you can see in the highlighted line, the /dev/md0 device has been created in the RAID 0 configuration using the /dev/sda and /dev/sdb devices.

Creating and Mounting the Filesystem

Next, create a filesystem on the array:

sudo mkfs.ext4 -F /dev/md0

Create a mount point to attach the new filesystem:

sudo mkdir -p /mnt/md0

You can mount the filesystem by typing:

sudo mount /dev/md0 /mnt/md0

Check whether the new space is available by typing:

df -h -x devtmpfs -x tmpfs

OutputFilesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G 1003M   23G   5% /
/dev/md0        196G   61M  186G   1% /mnt/md0

The new filesystem is mounted and accessible.

Saving the Array Layout

To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. You can automatically scan the active array and append the file by typing:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

sudo update-initramfs -u

Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Your RAID 0 array should now automatically be assembled and mounted each boot.

Creating a RAID 1 Array

The RAID 1 array type is implemented by mirroring data across all available disks. Each disk in a RAID 1 array gets a full copy of the data, providing redundancy in the event of a device failure.

Requirements: minimum of 2 storage devices
Primary benefit: Redundancy
Things to keep in mind: Since two copies of the data are maintained, only half of the disk space will be usable

Identifying the Component Devices

To get started, find the identifiers for the raw disks that you will be using:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE TYPE MOUNTPOINT
sda      100G        disk
sdb      100G        disk
vda       25G        disk 
├─vda1  24.9G ext4   part /
├─vda14    4M        part 
└─vda15  106M vfat   part /boot/efi

Creating the Array

To create a RAID 1 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

If the component devices you are using are not partitions with the boot flag enabled, you will likely see the following warning. It is safe to type y to continue:

Outputmdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 104792064K
Continue creating array? y

The mdadm tool will start to mirror the drives. This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

cat /proc/mdstat

OutputPersonalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb[1] sda[0]
      104792064 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  1.5% (1629632/104792064) finish=8.4min speed=203704K/sec

unused devices: <none>

As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 1 configuration using the /dev/sda and /dev/sdb devices. The second highlighted line shows the progress on the mirroring. You can continue the guide while this process completes.

Creating and Mounting the Filesystem

Next, create a filesystem on the array:

sudo mkfs.ext4 -F /dev/md0

Create a mount point to attach the new filesystem:

sudo mkdir -p /mnt/md0

You can mount the filesystem by typing:

sudo mount /dev/md0 /mnt/md0

Check whether the new space is available by typing:

df -h -x devtmpfs -x tmpfs

OutputFilesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G 1003M   23G   5% /
/dev/md0         98G   61M   93G   1% /mnt/md0

The new filesystem is mounted and accessible.

Saving the Array Layout

To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. You can automatically scan the active array and append the file by typing:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

sudo update-initramfs -u

Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Your RAID 1 array should now automatically be assembled and mounted each boot.

Creating a RAID 5 Array

The RAID 5 array type is implemented by striping data across the available devices. One component of each stripe is a calculated parity block. If a device fails, the parity block and the remaining blocks can be used to calculate the missing data. The device that receives the parity block is rotated so that each device has a balanced amount of parity information.

Requirements: minimum of 3 storage devices
Primary benefit: Redundancy with more usable capacity.
Things to keep in mind: While the parity information is distributed, one disk's worth of capacity will be used for parity. RAID 5 can suffer from very poor performance when in a degraded state.

Identifying the Component Devices

To get started, find the identifiers for the raw disks that you will be using:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE TYPE MOUNTPOINT
sda      100G        disk
sdb      100G        disk
sdc      100G        disk
vda       25G        disk 
├─vda1  24.9G ext4   part /
├─vda14    4M        part 
└─vda15  106M vfat   part /boot/efi

As you can see above, we have three disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda, /dev/sdb, and /dev/sdc identifiers for this session. These will be the raw components we will use to build the array.

Creating the Array

To create a RAID 5 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc

The mdadm tool will start to configure the array (it actually uses the recovery process to build the array for performance reasons). This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

cat /proc/mdstat

OutputPersonalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdc[3] sdb[1] sda[0]
      209584128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      [>....................]  recovery =  0.9% (1031612/104792064) finish=10.0min speed=171935K/sec

unused devices: <none>

As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 5 configuration using the /dev/sda, /dev/sdb and /dev/sdc devices. The second highlighted line shows the progress on the build.

Warning: Due to the way that mdadm builds RAID 5 arrays, while the array is still building, the number of spares in the array will be inaccurately reported. This means that you must wait for the array to finish assembling before updating the /etc/mdadm/mdadm.conf file. If you update the configuration file while the array is still building, the system will have incorrect information about the array state and will be unable to assemble it automatically at boot with the correct name.

You can continue the guide while this process completes.

Creating and Mounting the Filesystem

Next, create a filesystem on the array:

sudo mkfs.ext4 -F /dev/md0

Create a mount point to attach the new filesystem:

sudo mkdir -p /mnt/md0

You can mount the filesystem by typing:

sudo mount /dev/md0 /mnt/md0

Check whether the new space is available by typing:

df -h -x devtmpfs -x tmpfs

OutputFilesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G 1003M   23G   5% /
/dev/md0        196G   61M  186G   1% /mnt/md0

The new filesystem is mounted and accessible.

Saving the Array Layout

To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file.

As mentioned above, before you adjust the configuration, check again to make sure the array has finished assembling. Completing this step before the array is built will prevent the system from assembling the array correctly on reboot:

cat /proc/mdstat

OutputPersonalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdc[3] sdb[1] sda[0]
      209584128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

The output above shows that the rebuild is complete. Now, we can automatically scan the active array and append the file by typing:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

sudo update-initramfs -u

Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Your RAID 5 array should now automatically be assembled and mounted each boot.

Creating a RAID 6 Array

The RAID 6 array type is implemented by striping data across the available devices. Two components of each stripe are calculated parity blocks. If one or two devices fail, the parity blocks and the remaining blocks can be used to calculate the missing data. The devices that receive the parity blocks are rotated so that each device has a balanced amount of parity information. This is similar to a RAID 5 array, but allows for the failure of two drives.

Requirements: minimum of 4 storage devices
Primary benefit: Double redundancy with more usable capacity.
Things to keep in mind: While the parity information is distributed, two disk's worth of capacity will be used for parity. RAID 6 can suffer from very poor performance when in a degraded state.

Identifying the Component Devices

To get started, find the identifiers for the raw disks that you will be using:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE TYPE MOUNTPOINT
sda      100G        disk
sdb      100G        disk
sdc      100G        disk
sdd      100G        disk
vda       25G        disk 
├─vda1  24.9G ext4   part /
├─vda14    4M        part 
└─vda15  106M vfat   part /boot/efi

As you can see above, we have four disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd identifiers for this session. These will be the raw components we will use to build the array.

Creating the Array

To create a RAID 6 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

cat /proc/mdstat

OutputPersonalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid6 sdd[3] sdc[2] sdb[1] sda[0]
      209584128 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  resync =  0.3% (353056/104792064) finish=14.7min speed=117685K/sec

unused devices: <none>

As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 6 configuration using the /dev/sda, /dev/sdb, /dev/sdc and /dev/sdd devices. The second highlighted line shows the progress on the build. You can continue the guide while this process completes.

Creating and Mounting the Filesystem

Next, create a filesystem on the array:

sudo mkfs.ext4 -F /dev/md0

Create a mount point to attach the new filesystem:

sudo mkdir -p /mnt/md0

You can mount the filesystem by typing:

sudo mount /dev/md0 /mnt/md0

Check whether the new space is available by typing:

df -h -x devtmpfs -x tmpfs

OutputFilesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G 1003M   23G   5% /
/dev/md0        196G   61M  186G   1% /mnt/md0

The new filesystem is mounted and accessible.

Save the Array Layout

To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. We can automatically scan the active array and append the file by typing:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

sudo update-initramfs -u

Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Your RAID 6 array should now automatically be assembled and mounted each boot.

Creating a Complex RAID 10 Array

The RAID 10 array type is traditionally implemented by creating a striped RAID 0 array composed of sets of RAID 1 arrays. This nested array type gives both redundancy and high performance, at the expense of large amounts of disk space. The mdadm utility has its own RAID 10 type that provides the same type of benefits with increased flexibility. It is not created by nesting arrays, but has many of the same characteristics and guarantees. We will be using the mdadm RAID 10 here.

Requirements: minimum of 3 storage devices
Primary benefit: Performance and redundancy
Things to keep in mind: The amount of capacity reduction for the array is defined by the number of data copies you choose to keep. The number of copies that are stored with mdadm style RAID 10 is configurable.

By default, two copies of each data block will be stored in what is called the "near" layout. The possible layouts that dictate how each data block is stored are:

near: The default arrangement. Copies of each chunk are written consecutively when striping, meaning that the copies of the data blocks will be written around the same part of multiple disks.
far: The first and subsequent copies are written to different parts the storage devices in the array. For instance, the first chunk might be written near the beginning of a disk, while the second chunk would be written half way down on a different disk. This can give some read performance gains for traditional spinning disks at the expense of write performance.
offset: Each stripe is copied, offset by one drive. This means that the copies are offset from one another, but still close together on the disk. This helps minimize excessive seeking during some workloads.

You can find out more about these layouts by checking out the "RAID10" section of this man page:

man 4 md

You can also find this man page online here.

Identifying the Component Devices

To get started, find the identifiers for the raw disks that you will be using:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

OutputNAME     SIZE FSTYPE TYPE MOUNTPOINT
sda      100G        disk
sdb      100G        disk
sdc      100G        disk
sdd      100G        disk
vda       25G        disk 
├─vda1  24.9G ext4   part /
├─vda14    4M        part 
└─vda15  106M vfat   part /boot/efi

Creating the Array

To create a RAID 10 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices.

You can set up two copies using the near layout by not specifying a layout and copy number:

sudo mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

If you want to use a different layout, or change the number of copies, you will have to use the --layout= option, which takes a layout and copy identifier. The layouts are n for near, f for far, and o for offset. The number of copies to store is appended afterwards.

For instance, to create an array that has 3 copies in the offset layout, the command would look like this:

sudo mdadm --create --verbose /dev/md0 --level=10 --layout=o3 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

cat /proc/mdstat

OutputPersonalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid10 sdd[3] sdc[2] sdb[1] sda[0]
      209584128 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  1.3% (2832768/209584128) finish=15.8min speed=217905K/sec

unused devices: <none>

As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 10 configuration using the /dev/sda, /dev/sdb, /dev/sdc and /dev/sdd devices. The second highlighted area shows the layout that was used for this example (2 copies in the near configuration). The third highlighted area shows the progress on the build. You can continue the guide while this process completes.

Creating and Mounting the Filesystem

Next, create a filesystem on the array:

sudo mkfs.ext4 -F /dev/md0

Create a mount point to attach the new filesystem:

sudo mkdir -p /mnt/md0

You can mount the filesystem by typing:

sudo mount /dev/md0 /mnt/md0

Check whether the new space is available by typing:

df -h -x devtmpfs -x tmpfs

OutputFilesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G 1003M   23G   5% /
/dev/md0        196G   61M  186G   1% /mnt/md0

The new filesystem is mounted and accessible.

Saving the Array Layout

To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. We can automatically scan the active array and append the file by typing:

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

sudo update-initramfs -u

Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Your RAID 10 array should now automatically be assembled and mounted each boot.

Conclusion

In this guide, we demonstrated how to create various types of arrays using Linux's mdadm software RAID utility. RAID arrays offer some compelling redundancy and performance enhancements over using multiple disks individually.

Once you have settled on the type of array needed for your environment and created the device, you will need to learn how to perform day-to-day management with mdadm. Our guide on how to manage RAID arrays with mdadm can help get you started.

How To Install the Apache Web Server on Debian 9

2018-09-05T19:48:01Z

Introduction

The Apache HTTP server is the most widely-used web server in the world. It provides many powerful features including dynamically loadable modules, robust media support, and extensive integration with other popular software.

In this guide, we'll explain how to install an Apache web server on your Debian 9 server.

Prerequisites

Before you begin this guide, you should have a regular, non-root user with sudo privileges configured on your server. Additionally, you will need to enable a basic firewall to block non-essential ports. You can learn how to configure a regular user account and set up a firewall for your server by following our initial server setup guide for Debian 9.

When you have an account available, log in as your non-root user to begin.

Step 1 — Installing Apache

Apache is available within Debian's default software repositories, making it possible to install it using conventional package management tools.

Let's begin by updating the local package index to reflect the latest upstream changes:

sudo apt update

Then, install the apache2 package:

sudo apt install apache2

After confirming the installation, apt will install Apache and all required dependencies.

Step 2 — Adjusting the Firewall

Before testing Apache, it's necessary to modify the firewall settings to allow outside access to the default web ports. Assuming that you followed the instructions in the prerequisites, you should have a UFW firewall configured to restrict access to your server.

During installation, Apache registers itself with UFW to provide a few application profiles that can be used to enable or disable access to Apache through the firewall.

List the ufw application profiles by typing:

sudo ufw app list

You will see a list of the application profiles:

OutputAvailable applications:
  AIM
  Bonjour
  CIFS
. . . 
 WWW
 WWW Cache
 WWW Full
 WWW Secure
. . .

The Apache profiles begin with WWW:

WWW: This profile opens only port 80 (normal, unencrypted web traffic)
WWW Cache: This profile opens only port 8080 (sometimes used for caching and web proxies)
WWW Full: This profile opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
WWW Secure: This profile opens only port 443 (TLS/SSL encrypted traffic)

It is recommended that you enable the most restrictive profile that will still allow the traffic you've configured. Since we haven't configured SSL for our server yet in this guide, we will only need to allow traffic on port 80:

sudo ufw allow 'WWW'

You can verify the change by typing:

sudo ufw status

You should see HTTP traffic allowed in the displayed output:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
WWW                        ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
WWW (v6)                   ALLOW       Anywhere (v6)

As you can see, the profile has been activated to allow access to the web server.

Step 3 — Checking your Web Server

At the end of the installation process, Debian 9 starts Apache. The web server should already be up and running.

Check with the systemd init system to make sure the service is running by typing:

sudo systemctl status apache2

Output● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 19:21:48 UTC; 13min ago
 Main PID: 12849 (apache2)
   CGroup: /system.slice/apache2.service
           ├─12849 /usr/sbin/apache2 -k start
           ├─12850 /usr/sbin/apache2 -k start
           └─12852 /usr/sbin/apache2 -k start

Sep 05 19:21:48 apache systemd[1]: Starting The Apache HTTP Server...
Sep 05 19:21:48 apache systemd[1]: Started The Apache HTTP Server.

As you can see from this output, the service appears to have started successfully. However, the best way to test this is to request a page from Apache.

You can access the default Apache landing page to confirm that the software is running properly through your IP address. If you do not know your server's IP address, you can get it a few different ways from the command line.

Try typing this at your server's command prompt:

hostname -I

You will get back a few addresses separated by spaces. You can try each in your web browser to see if they work.

An alternative is using the curl tool, which should give you your public IP address as seen from another location on the internet.

First, install curl using apt:

sudo apt install curl

Then, use curl to retrieve icanhazip.com using IPv4:

curl -4 icanhazip.com

When you have your server's IP address, enter it into your browser's address bar:

http://your_server_ip

You should see the default Debian 9 Apache web page:

This page indicates that Apache is working correctly. It also includes some basic information about important Apache files and directory locations.

Step 4 — Managing the Apache Process

Now that you have your web server up and running, let's go over some basic management commands.

To stop your web server, type:

sudo systemctl stop apache2

To start the web server when it is stopped, type:

sudo systemctl start apache2

To stop and then start the service again, type:

sudo systemctl restart apache2

If you are simply making configuration changes, Apache can often reload without dropping connections. To do this, use this command:

sudo systemctl reload apache2

By default, Apache is configured to start automatically when the server boots. If this is not what you want, disable this behavior by typing:

sudo systemctl disable apache2

To re-enable the service to start up at boot, type:

sudo systemctl enable apache2

Apache should now start automatically when the server boots again.

Step 5 — Setting Up Virtual Hosts (Recommended)

When using the Apache web server, you can use virtual hosts (similar to server blocks in Nginx) to encapsulate configuration details and host more than one domain from a single server. We will set up a domain called example.com, but you should replace this with your own domain name. To learn more about setting up a domain name with DigitalOcean, see our Introduction to DigitalOcean DNS.

Apache on Debian 9 has one server block enabled by default that is configured to serve documents from the /var/www/html directory. While this works well for a single site, it can become unwieldy if you are hosting multiple sites. Instead of modifying /var/www/html, let's create a directory structure within /var/www for our example.com site, leaving /var/www/html in place as the default directory to be served if a client request doesn't match any other sites.

Create the directory for example.com as follows, using the -p flag to create any necessary parent directories:

sudo mkdir -p /var/www/example.com/html

Next, assign ownership of the directory with the $USER environmental variable:

sudo chown -R $USER:$USER /var/www/example.com/html

The permissions of your web roots should be correct if you haven't modified your unmask value, but you can make sure by typing:

sudo chmod -R 755 /var/www/example.com

Next, create a sample index.html page using nano or your favorite editor:

nano /var/www/example.com/html/index.html

Inside, add the following sample HTML:

/var/www/example.com/html/index.html

<html>
    <head>
        <title>Welcome to Example.com!</title>
    </head>
    <body>
        <h1>Success!  The example.com server block is working!</h1>
    </body>
</html>

Save and close the file when you are finished.

In order for Apache to serve this content, it's necessary to create a virtual host file with the correct directives. Instead of modifying the default configuration file located at /etc/apache2/sites-available/000-default.conf directly, let's make a new one at /etc/apache2/sites-available/example.com.conf:

sudo nano /etc/apache2/sites-available/example.com.conf

Paste in the following configuration block, which is similar to the default, but updated for our new directory and domain name:

/etc/apache2/sites-available/example.com.conf

<VirtualHost *:80>
    ServerAdmin admin@example.com
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/example.com/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Notice that we've updated the DocumentRoot to our new directory and ServerAdmin to an email that the example.com site administrator can access. We've also added two directives: ServerName, which establishes the base domain that should match for this virtual host definition, and ServerAlias, which defines further names that should match as if they were the base name.

Save and close the file when you are finished.

Let's enable the file with the a2ensite tool:

sudo a2ensite example.com.conf

Disable the default site defined in 000-default.conf:

sudo a2dissite 000-default.conf

Next, let's test for configuration errors:

sudo apache2ctl configtest

You should see the following output:

OutputSyntax OK

Restart Apache to implement your changes:

sudo systemctl restart apache2

Apache should now be serving your domain name. You can test this by navigating to http://example.com, where you should see something like this:

Step 6 – Getting Familiar with Important Apache Files and Directories

Now that you know how to manage the Apache service itself, you should take a few minutes to familiarize yourself with a few important directories and files.

Content

/var/www/html: The actual web content, which by default only consists of the default Apache page you saw earlier, is served out of the /var/www/html directory. This can be changed by altering Apache configuration files.

Server Configuration

/etc/apache2: The Apache configuration directory. All of the Apache configuration files reside here.
/etc/apache2/apache2.conf: The main Apache configuration file. This can be modified to make changes to the Apache global configuration. This file is responsible for loading many of the other files in the configuration directory.
/etc/apache2/ports.conf: This file specifies the ports that Apache will listen on. By default, Apache listens on port 80 and additionally listens on port 443 when a module providing SSL capabilities is enabled.
/etc/apache2/sites-available/: The directory where per-site virtual hosts can be stored. Apache will not use the configuration files found in this directory unless they are linked to the sites-enabled directory. Typically, all server block configuration is done in this directory, and then enabled by linking to the other directory with the a2ensite command.
/etc/apache2/sites-enabled/: The directory where enabled per-site virtual hosts are stored. Typically, these are created by linking to configuration files found in the sites-available directory with the a2ensite. Apache reads the configuration files and links found in this directory when it starts or reloads to compile a complete configuration.
/etc/apache2/conf-available/, /etc/apache2/conf-enabled/: These directories have the same relationship as the sites-available and sites-enabled directories, but are used to store configuration fragments that do not belong in a virtual host. Files in the conf-available directory can be enabled with the a2enconf command and disabled with the a2disconf command.
/etc/apache2/mods-available/, /etc/apache2/mods-enabled/: These directories contain the available and enabled modules, respectively. Files in ending in .load contain fragments to load specific modules, while files ending in .conf contain the configuration for those modules. Modules can be enabled and disabled using the a2enmod and a2dismod command.

Server Logs

/var/log/apache2/access.log: By default, every request to your web server is recorded in this log file unless Apache is configured to do otherwise.
/var/log/apache2/error.log: By default, all errors are recorded in this file. The LogLevel directive in the Apache configuration specifies how much detail the error logs will contain.

Conclusion

Now that you have your web server installed, you have many options for the type of content you can serve and the technologies you can use to create a richer experience.

If you'd like to build out a more complete application stack, you can look at this article on how to configure a LAMP stack on Debian 9.

How to Install MongoDB on Debian 9

2018-09-05T19:43:46Z

Introduction

MongoDB is a free and open-source NoSQL document database used commonly in modern web applications.

In this tutorial you will install MongoDB, manage its service, and optionally enable remote access.

Prerequisites

To follow this tutorial, you will need one Debian 9 server set up by following this initial server setup tutorial, including a sudo-enabled non-root user and a firewall.

Step 1 — Installing MongoDB

Debian 9's official package repositories include a slightly-out-of-date version of MongoDB, which means we'll install from the official MongoDB repo instead.

First, we need to add the MongoDB signing key with apt-key add. We'll need to make sure the curl command is installed before doing so:

sudo apt install curl

Next we download the key and pass it to apt-key add:

curl https://www.mongodb.org/static/pgp/server-4.0.asc | sudo apt-key add -

Next we'll create a source list for the MongoDB repo, so apt knows where to download from. First open the source list file in a text editor:

sudo nano /etc/apt/sources.list.d/mongodb-org-4.0.list

This will open a new blank file. Paste in the following:

/etc/apt/sources.list.d/mongodb-org-4.0.list

deb http://repo.mongodb.org/apt/debian stretch/mongodb-org/4.0 main

Save and close the file, then update your package cache:

sudo apt update

Install the mongodb-org package to install the server and some supporting tools:

sudo apt-get install mongodb-org

Finally, enable and start the mongod service to get your MongoDB database running:

sudo systemctl enable mongod
sudo systemctl start mongod

We've now installed and started the latest stable version of MongoDB, along with helpful management tools for the MongoDB server.

Next, let's verify that the server is running and works correctly.

Step 2 — Checking the Service and Database

We started MongoDB service in the previous step, now let's verify that it is started and the database is working.

First, check the service's status:

sudo systemctl status mongod

You'll see this output:

Output
● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 16:59:56 UTC; 3s ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 4321 (mongod)
    Tasks: 26
   CGroup: /system.slice/mongod.service
           └─4321 /usr/bin/mongod --config /etc/mongod.conf

According to systemd, the MongoDB server is up and running.

We can verify this further by actually connecting to the database server and executing a diagnostic command

Execute this command:

mongo --eval 'db.runCommand({ connectionStatus: 1 })'

This will output the current database version, the server address and port, and the output of the status command:

OutputMongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.0.2
{
    "authInfo" : {
        "authenticatedUsers" : [ ],
        "authenticatedUserRoles" : [ ]
    },
    "ok" : 1
}

A value of 1 for the ok field in the response indicates that the server is working properly.

Next, we'll look at how to manage the server instance.

Step 3 — Managing the MongoDB Service

MongoDB installs as a systemd service, which means that you can manage it using standard systemd commands alongside all other system services in Ubuntu.

To verify the status of the service, type:

sudo systemctl status mongod

You can stop the server anytime by typing:

sudo systemctl stop mongod

To start the server when it is stopped, type:

sudo systemctl start mongod

You can also restart the server with a single command:

sudo systemctl restart mongod

In the previous step we enabled MongoDB to start automatically with the server. If you wish to disable the automatic startup, type:

sudo systemctl disable mongod

It's just as easy to enable it again. To do this, use:

sudo systemctl enable mongod

Next, let's adjust the firewall settings for our MongoDB installation.

Step 4 — Adjusting the Firewall (Optional)

Assuming you have followed the initial server setup tutorial instructions to enable the firewall on your server, the MongoDB server will be inaccessible from the internet.

If you intend to use the MongoDB server only locally with applications running on the same server, this is the recommended and secure setting. However, if you would like to be able to connect to your MongoDB server from the internet, you have to allow the incoming connections in ufw.

To allow access to MongoDB on its default port 27017 from everywhere, you could use sudo ufw allow 27017. However, enabling internet access to MongoDB server on a default installation gives anyone unrestricted access to the database server and its data.

In most cases, MongoDB should be accessed only from certain trusted locations, such as another server hosting an application. To accomplish this task, you can allow access on MongoDB's default port while specifying the IP address of another server that will be explicitly allowed to connect:

sudo ufw allow from your_other_server_ip/32 to any port 27017

You can verify the change in firewall settings with ufw:

sudo ufw status

You should see traffic to port 27017 allowed in the output:

Output

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
27017                      ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
27017 (v6)                 ALLOW       Anywhere (v6)

If you have decided to allow only a certain IP address to connect to MongoDB server, the IP address of the allowed location will be listed instead of Anywhere in the output.

You can find more advanced firewall settings for restricting access to services in UFW Essentials: Common Firewall Rules and Commands.

Even though the port is open, MongoDB is currently only listening on the local address 127.0.0.1. To allow remote connections, add your server's publicly-routable IP address to the mongod.conf file.

Open the MongoDB configuration file in your editor:

sudo nano /etc/mongod.conf

Add your server's IP address to the bindIP value:

/etc/mongod.conf

. . .
# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1,your_server_ip
. . .

Be sure to place a comma between the existing IP address and the one you added.

Save the file, exit the editor, and restart MongoDB:

sudo systemctl restart mongod

MongoDB is now listening for remote connections, but anyone can access it. Follow Part 2 of How to Install and Secure MongoDB on Ubuntu 16.04 to add an administrative user and lock things down further.

Conclusion

You can find more in-depth tutorials on how to configure and use MongoDB in these DigitalOcean community articles. The official MongoDB documentation is also a great resource on the possibilities that MongoDB provides.

How To Secure Nginx with Let's Encrypt on Debian 9

2018-09-05T17:30:09Z

Introduction

Let's Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free TLS/SSL certificates, enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx.

In this tutorial, you will use Certbot to obtain a free SSL certificate for Nginx on Debian 9 and set up your certificate to renew automatically.

This tutorial will use a separate Nginx server block file instead of the default file. We recommend creating new Nginx server block files for each domain because it helps to avoid common mistakes and maintains the default files as a fallback configuration.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server, set up by following this initial server setup for Debian 9 tutorial, along with a sudo non-root user and a firewall.
A fully registered domain name. This tutorial will use example.com throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.
Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.
- An A record with example.com pointing to your server's public IP address.
- An A record with www.example.com pointing to your server's public IP address.
Nginx installed by following How To Install Nginx on Debian 9. Be sure that you have a server block for your domain. This tutorial will use /etc/nginx/sites-available/example.com as an example.

Step 1 — Installing Certbot

The first step to using Let's Encrypt to obtain an SSL certificate is to install the Certbot software on your server.

Certbot is in very active development, so the Certbot packages provided by Debian with current stable releases tend to be outdated. However, we can obtain a more up-to-date package by enabling the Debian 9 backports repository in /etc/apt/sources.list, where the apt package manager looks for package sources. The backports repository includes recompiled packages that can be run without new libraries on stable Debian distributions.

To add the backports repository, first open /etc/apt/sources.list:

sudo nano /etc/apt/sources.list

At the bottom of the file, add the following mirrors from the Debian project:

/etc/apt/sources.list

...
deb http://deb.debian.org/debian stretch-backports main contrib non-free
deb-src http://deb.debian.org/debian stretch-backports main contrib non-free

This includes the main packages, which are Debian Free Software Guidelines (DFSG)- compliant, as well as the non-free and contrib components, which are either not DFSG-compliant themselves or include dependencies in this category.

Save and close the file when you are finished.

Update the package list to pick up the new repository's package information:

sudo apt update

And finally, install Certbot's Nginx package with apt:

sudo apt install python-certbot-nginx -t stretch-backports

Certbot is now ready to use, but in order for it to configure SSL for Nginx, we need to verify some of Nginx's configuration.

Step 2 — Confirming Nginx's Configuration

Certbot needs to be able to find the correct server block in your Nginx configuration for it to be able to automatically configure SSL. Specifically, it does this by looking for a server_name directive that matches your requested domain.

If you followed the server block setup step in the Nginx installation tutorial, you should have a server block for your domain at /etc/nginx/sites-available/example.com with the server_name directive already set appropriately.

To check, open the server block file for your domain using nano or your favorite text editor:

sudo nano /etc/nginx/sites-available/example.com

Find the existing server_name line. It should look like this:

/etc/nginx/sites-available/example.com

...
server_name example.com www.example.com;
...

If it does, exit your editor and move on to the next step.

If it doesn't, update it to match. Then save the file, quit your editor, and verify the syntax of your configuration edits:

sudo nginx -t

If you get an error, reopen the server block file and check for any typos or missing characters. Once your configuration file syntax is correct, reload Nginx to load the new configuration:

sudo systemctl reload nginx

Certbot can now find the correct server block and update it.

Next, let's update the firewall to allow HTTPS traffic.

Step 3 — Allowing HTTPS Through the Firewall

If you have the ufw firewall enabled, as recommended in the prerequisite guides, you'll need to adjust the settings to allow for HTTPS traffic.

You can see the current setting by typing:

sudo ufw status

It will probably look like this, meaning that only HTTP traffic is allowed to the web server:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Nginx HTTP                 ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

To let in HTTPS traffic, allow the Nginx Full profile and delete the redundant Nginx HTTP profile allowance:

sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'

Your status should now look like this:

sudo ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

Next, let's run Certbot and fetch our certificates.

Step 4 — Obtaining an SSL Certificate

Certbot provides a variety of ways to obtain SSL certificates through plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary. To use this plugin, type the following:

sudo certbot --nginx -d example.com -d www.example.com

This runs certbot with the --nginx plugin, using -d to specify the names we'd like the certificate to be valid for.

If that's successful, certbot will ask how you'd like to configure your HTTPS settings.

OutputPlease choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Select your choice then hit ENTER. The configuration will be updated, and Nginx will reload to pick up the new settings. certbot will wrap up with a message telling you the process was successful and where your certificates are stored:

OutputIMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2018-07-23. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Let's finish by testing the renewal process.

Step 5 — Verifying Certbot Auto-Renewal

To test the renewal process, you can do a dry run with certbot:

sudo certbot renew --dry-run

If you see no errors, you're all set. When necessary, Certbot will renew your certificates and reload Nginx to pick up the changes. If the automated renewal process ever fails, Let’s Encrypt will send a message to the email you specified, warning you when your certificate is about to expire.

Conclusion

In this tutorial, you installed the Let's Encrypt client certbot, downloaded SSL certificates for your domain, configured Nginx to use these certificates, and set up automatic certificate renewal. If you have further questions about using Certbot, their documentation is a good place to start.

How to Install and Configure VNC on Debian 9

2018-09-05T17:42:39Z

Introduction

Virtual Network Computing, or VNC, is a connection system that allows you to use your keyboard and mouse to interact with a graphical desktop environment on a remote server. It makes managing files, software, and settings on a remote server easier for users who are not yet comfortable with the command line.

In this guide, you'll set up a VNC server on a Debian 9 server and connect to it securely through an SSH tunnel. You'll use TightVNC, a fast and lightweight remote control package. This choice will ensure that our VNC connection will be smooth and stable even on slower internet connections.

Prerequisites

To complete this tutorial, you'll need:

One Debian 9 server set up by following the Debian 9 initial server setup guide, including a non-root user with sudo access and a firewall.
A local computer with a VNC client installed that supports VNC connections over SSH tunnels.
- On Winows, you can use TightVNC, RealVNC, or UltraVNC.
- On macOS, you can use the built-in Screen Sharing program, or can use a cross-platform app like RealVNC.
- On Linux, you can choose from many options, including vinagre, krdc, RealVNC, or TightVNC.

Step 1 — Installing the Desktop Environment and VNC Server

By default, a Debian 9 server does not come with a graphical desktop environment or a VNC server installed, so we'll begin by installing those. Specifically, we will install packages for the latest Xfce desktop environment and the TightVNC package available in the official Debian repository.

On your server, update your list of packages:

sudo apt update

Now install the Xfce desktop environment on your server:

sudo apt install xfce4 xfce4-goodies

During the installation, you'll be prompted to select your keyboard layout from a list of possible options. Choose the one that's appropriate for your language and press Enter. The installation will continue.

Once that installation completes, install the TightVNC server:

sudo apt install tightvncserver

To complete the VNC server's initial configuration after installation, use the vncserver command to set up a secure password and create the initial configuration files:

vncserver

You'll be prompted to enter and verify a password to access your machine remotely:

OutputYou will require a password to access your desktops.

Password:
Verify:

The password must be between six and eight characters long. Passwords more than 8 characters will be truncated automatically.

Once you verify the password, you'll have the option to create a a view-only password. Users who log in with the view-only password will not be able to control the VNC instance with their mouse or keyboard. This is a helpful option if you want to demonstrate something to other people using your VNC server, but this isn't required.

The process then creates the necessary default configuration files and connection information for the server:

Output
Would you like to enter a view-only password (y/n)? n
xauth:  file /home/sammy/.Xauthority does not exist

New 'X' desktop is your_hostname:1

Creating default startup script /home/sammy/.vnc/xstartup
Starting applications specified in /home/sammy/.vnc/xstartup
Log file is /home/sammy/.vnc/your_hostname:1.log

Now let's configure the VNC server.

Step 2 — Configuring the VNC Server

The VNC server needs to know which commands to execute when it starts up. Specifically, VNC needs to know which graphical desktop it should connect to.

These commands are located in a configuration file called xstartup in the .vnc folder under your home directory. The startup script was created when you ran the vncserver in the previous step, but we'll create our own to launch the Xfce desktop.

When VNC is first set up, it launches a default server instance on port 5901. This port is called a display port, and is referred to by VNC as :1. VNC can launch multiple instances on other display ports, like :2, :3, and so on.

Because we are going to be changing how the VNC server is configured, first stop the VNC server instance that is running on port 5901 with the following command:

vncserver -kill :1

The output should look like this, although you'll see a different PID:

Output
Killing Xtightvnc process ID 17648

Before you modify the xstartup file, back up the original:

mv ~/.vnc/xstartup ~/.vnc/xstartup.bak

Now create a new xstartup file and open it in your text editor:

nano ~/.vnc/xstartup

Commands in this file are executed automatically whenever you start or restart the VNC server. We need VNC to start our desktop environment if it's not already started. Add these commands to the file:

~/.vnc/xstartup#!/bin/bash
xrdb $HOME/.Xresources
startxfce4 &

The first command in the file, xrdb $HOME/.Xresources, tells VNC's GUI framework to read the server user's .Xresources file. .Xresources is where a user can make changes to certain settings of the graphical desktop, like terminal colors, cursor themes, and font rendering. The second command tells the server to launch Xfce, which is where you will find all of the graphical software that you need to comfortably manage your server.

To ensure that the VNC server will be able to use this new startup file properly, we'll need to make it executable.

sudo chmod +x ~/.vnc/xstartup

Now, restart the VNC server.

vncserver

You'll see output similar to this:

Output
New 'X' desktop is your_hostname:1

Starting applications specified in /home/sammy/.vnc/xstartup
Log file is /home/sammy/.vnc/your_hostname:1.log

With the configuration in place, let's connect to the server from our local machine.

Step 3 — Connecting the VNC Desktop Securely

VNC itself doesn't use secure protocols when connecting. We'll use an SSH tunnel to connect securely to our server, and then tell our VNC client to use that tunnel rather than making a direct connection.

Create an SSH connection on your local computer that securely forwards to the localhost connection for VNC. You can do this via the terminal on Linux or macOS with the following command:

ssh -L 5901:127.0.0.1:5901 -C -N -l sammy your_server_ip

The -L switch specifies the port bindings. In this case we're binding port 5901 of the remote connection to port 5901 on your local machine. The -C switch enables compression, while the -N switch tells ssh that we don't want to execute a remote command. The -l switch specifies the remote login name.

Remember to replace sammy and your_server_ip with the sudo non-root username and IP address of your server.

If you are using a graphical SSH client, like PuTTY, use your_server_ip as the connection IP, and set localhost:5901 as a new forwarded port in the program's SSH tunnel settings.

Once the tunnel is running, use a VNC client to connect to localhost:5901. You'll be prompted to authenticate using the password you set in Step 1.

Once you are connected, you'll see the default Xfce desktop.

Select Use default config to configure your desktop quickly.

You can access files in your home directory with the file manager or from the command line, as seen here:

On your local machine, press CTRL+C in your terminal to stop the SSH tunnel and return to your prompt. This will disconnect your VNC session as well.

Next let's set up the VNC server as a service.

Step 4 — Running VNC as a System Service

Next, we'll set up the VNC server as a systemd service so we can start, stop, and restart it as needed, like any other service. This will also ensure that VNC starts up when your server reboots.

First, create a new unit file called /etc/systemd/system/vncserver@.service using your favorite text editor:

sudo nano /etc/systemd/system/vncserver@.service

The @ symbol at the end of the name will let us pass in an argument we can use in the service configuration. We'll use this to specify the VNC display port we want to use when we manage the service.

Add the following lines to the file. Be sure to change the value of User, Group, WorkingDirectory, and the username in the value of PIDFILE to match your username:

/etc/systemd/system/vncserver@.service [Unit]
Description=Start TightVNC server at startup
After=syslog.target network.target

[Service]
Type=forking
User=sammy
Group=sammy
WorkingDirectory=/home/sammy

PIDFile=/home/sammy/.vnc/%H:%i.pid
ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1
ExecStart=/usr/bin/vncserver -depth 24 -geometry 1280x800 :%i
ExecStop=/usr/bin/vncserver -kill :%i

[Install]
WantedBy=multi-user.target

The ExecStartPre command stops VNC if it's already running. The ExecStart command starts VNC and sets the color depth to 24-bit color with a resolution of 1280x800. You can modify these startup options as well to meet your needs.

Save and close the file.

Next, make the system aware of the new unit file.

sudo systemctl daemon-reload

Enable the unit file.

sudo systemctl enable vncserver@1.service

The 1 following the @ sign signifies which display number the service should appear over, in this case the default :1 as was discussed in Step 2..

Stop the current instance of the VNC server if it's still running.

vncserver -kill :1

Then start it as you would start any other systemd service.

sudo systemctl start vncserver@1

You can verify that it started with this command:

sudo systemctl status vncserver@1

If it started correctly, the output should look like this:

Output● vncserver@1.service - Start TightVNC server at startup
   Loaded: loaded (/etc/systemd/system/vncserver@.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 16:47:40 UTC; 3s ago
  Process: 4977 ExecStart=/usr/bin/vncserver -depth 24 -geometry 1280x800 :1 (code=exited, status=0/SUCCESS)
  Process: 4971 ExecStartPre=/usr/bin/vncserver -kill :1 > /dev/null 2>&1 (code=exited, status=0/SUCCESS)
 Main PID: 4987 (Xtightvnc)

...

Your VNC server will now be available when you reboot the machine.

Start your SSH tunnel again:

ssh -L 5901:127.0.0.1:5901 -C -N -l sammy your_server_ip

Then make a new connection using your VNC client software to localhost:5901 to connect to your machine.

Conclusion

You now have a secured VNC server up and running on your Debian 9 server. Now you'll be able to manage your files, software, and settings with an easy-to-use and familiar graphical interface, and you'll be able to run graphical software like web browsers remotely.

How To Install the Latest MySQL on Debian 9

2018-09-05T16:30:26Z

Introduction

MySQL is a prominent open source database management system used to store and retrieve data for a wide variety of popular applications. MySQL is the M in the LAMP stack, a commonly used set of open source software that also includes Linux, the Apache web server, and the PHP programming language.

In Debian 9, MariaDB, a community fork of the MySQL project, is packaged as the default MySQL variant. While, MariaDB works well in most cases, if you need features found only in Oracle's MySQL, you can install and use packages from a repository maintained by the MySQL developers.

To install the latest version of MySQL, we'll add this repository, install the MySQL software itself, secure the install, and finally we'll test that MySQL is running and responding to commands.

Prerequisites

Before starting this tutorial, you will need:

One Debian 9 server set up by following this initial server setup guide, including a non-root user with sudo privileges and a firewall.

Step 1 — Adding the MySQL Software Repository

The MySQL developers provide a .deb package that handles configuring and installing the official MySQL software repositories. Once the repositories are set up, we'll be able to use Ubuntu's standard apt command to install the software. We'll download this .deb file with wget and then install it with the dpkg command.

First, load the MySQL download page in your web browser. Find the Download button in the lower-right corner and click through to the next page. This page will prompt you to log in or sign up for an Oracle web account. We can skip that and instead look for the link that says No thanks, just start my download. Right-click the link and select Copy Link Address (this option may be worded differently, depending on your browser).

Now we're going to download the file. On your server, move to a directory you can write to. Download the file using wget, remembering to paste the address you just copied in place of the highlighted portion below:

cd /tmp
wget https://dev.mysql.com/get/mysql-apt-config_0.8.10-1_all.deb

The file should now be downloaded in our current directory. List the files to make sure:

ls

You should see the filename listed:

Output
mysql-apt-config_0.8.10-1_all.deb
. . .

Now we're ready to install:

sudo dpkg -i mysql-apt-config*

dpkg is used to install, remove, and inspect .deb software packages. The -i flag indicates that we'd like to install from the specified file.

During the installation, you'll be presented with a configuration screen where you can specify which version of MySQL you'd prefer, along with an option to install repositories for other MySQL-related tools. The defaults will add the repository information for the latest stable version of MySQL and nothing else. This is what we want, so use the down arrow to navigate to the Ok menu option and hit ENTER.

The package will now finish adding the repository. Refresh your apt package cache to make the new software packages available:

sudo apt update

Now that we've added the MySQL repositories, we're ready to install the actual MySQL server software. If you ever need to update the configuration of these repositories, just run sudo dpkg-reconfigure mysql-apt-config, select new options, and then sudo apt-get update to refresh your package cache.

Step 2 — Installing MySQL

Having added the repository and with our package cache freshly updated, we can now use apt to install the latest MySQL server package:

sudo apt install mysql-server

apt will look at all available mysql-server packages and determine that the MySQL provided package is the newest and best candidate. It will then calculate package dependencies and ask you to approve the installation. Type y then ENTER. The software will install.

You will be asked to set a root password during the configuration phase of the installation. Choose and confirm a secure password to continue. Next, a prompt will appear asking for you to select a default authentication plugin. Read the display to understand the choices. If you are not sure, choosing Use Strong Password Encryption is safer.

MySQL should be installed and running now. Let's check using systemctl:

sudo systemctl status mysql

Output● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-09-05 15:58:21 UTC; 30s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 12805 (mysqld)
   Status: "SERVER_OPERATING"
   CGroup: /system.slice/mysql.service
           └─12805 /usr/sbin/mysqld

Sep 05 15:58:15 mysql1 systemd[1]: Starting MySQL Community Server...
Sep 05 15:58:21 mysql1 systemd[1]: Started MySQL Community Server.

The Active: active (running) line means MySQL is installed and running. Now we'll make the installation a little more secure.

Step 3 — Securing MySQL

MySQL comes with a command we can use to perform a few security-related updates on our new install. Let's run it now:

mysql_secure_installation

This will ask you for the MySQL root password that you set during installation. Type it in and press ENTER. Now we'll answer a series of yes or no prompts. Let's go through them:

First, we are asked about the validate password plugin, a plugin that can automatically enforce certain password strength rules for your MySQL users. Enabling this is a decision you'll need to make based on your individual security needs. Type y and ENTER to enable it, or just hit ENTER to skip it. If enabled, you will also be prompted to choose a level from 0–2 for how strict the password validation will be. Choose a number and hit ENTER to continue.

Next you'll be asked if you want to change the root password. Since we just created the password when we installed MySQL, we can safely skip this. Hit ENTER to continue without updating the password.

The rest of the prompts can be answered yes. You will be asked about removing the anonymous MySQL user, disallowing remote root login, removing the test database, and reloading privilege tables to ensure the previous changes take effect properly. These are all a good idea. Type y and hit ENTER for each.

The script will exit after all the prompts are answered. Now our MySQL installation is reasonably secured. Let's test it again by running a client that connects to the server and returns some information.

Step 4 – Testing MySQL

mysqladmin is a command line administrative client for MySQL. We'll use it to connect to the server and output some version and status information:

mysqladmin -u root -p version

The -u root portion tells mysqladmin to log in as the MySQL root user, -p instructs the client to ask for a password, and version is the actual command we want to run.

The output will let us know what version of the MySQL server is running, its uptime, and some other status information:

Outputmysqladmin  Ver 8.0.12 for Linux on x86_64 (MySQL Community Server - GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Server version      8.0.12
Protocol version    10
Connection      Localhost via UNIX socket
UNIX socket     /var/run/mysqld/mysqld.sock
Uptime:         6 min 42 sec

Threads: 2  Questions: 12  Slow queries: 0  Opens: 123  Flush tables: 2  Open tables: 99  Queries per second avg: 0.029

If you received similar output, congrats! You've successfully installed the latest MySQL server and secured it.

Conclusion

You've now completed a basic install of the latest version of MySQL, which should work for many popular applications.

How To Install Python 3 and Set Up a Programming Environment on Debian 9

2018-09-04T22:12:59Z

Introduction

A flexible and versatile programming language, Python is effective for many use cases, including scripting, automation, data analysis, machine learning, and back-end development. First published in 1991 with a name inspired by the British comedy group Monty Python, the development team wanted to make Python a language that was fun to use. Quick to set up, and written in a relatively straightforward style with immediate feedback on errors, Python is a great choice for beginners and experienced developers alike. Python 3 is the most current version of the language and is considered to be the future of Python.

This tutorial will get your Debian 9 server set up with a Python 3 programming environment. Programming on a server has many advantages and supports collaboration across development projects.

Prerequisites

In order to complete this tutorial, you should have a non-root user with sudo privileges on a Debian 9 server. To learn how to achieve this setup, follow our Debian 9 initial server setup guide.

If you’re not already familiar with a terminal environment, you may find the article “An Introduction to the Linux Terminal” useful for becoming better oriented with the terminal.

With your server and user set up, you are ready to begin.

Step 1 — Setting Up Python 3

Debian Linux ships with both Python 3 and Python 2 pre-installed. To make sure that our versions are up-to-date, let’s update and upgrade the system with the apt command to work with the Advanced Packaging Tool:

sudo apt update
sudo apt -y upgrade

The -y flag will confirm that we are agreeing for all items to be installed.

Once the process is complete, we can check the version of Python 3 that is installed in the system by typing:

python3 -V

You’ll receive output in the terminal window that will let you know the version number. While this number may vary, the output will be similar to this:

Output
Python 3.5.3

To manage software packages for Python, let’s install pip, a tool that will install and manage programming packages we may want to use in our development projects. You can learn more about modules or packages that you can install with pip by reading “How To Import Modules in Python 3.”

sudo apt install -y python3-pip

Python packages can be installed by typing:

pip3 install package_name

Here, package_name can refer to any Python package or library, such as Django for web development or NumPy for scientific computing. So if you would like to install NumPy, you can do so with the command pip3 install numpy.

There are a few more packages and development tools to install to ensure that we have a robust set-up for our programming environment:

sudo apt install build-essential libssl-dev libffi-dev python3-dev

Once Python is set up, and pip and other tools are installed, we can set up a virtual environment for our development projects.

Step 2 — Setting Up a Virtual Environment

Virtual environments enable you to have an isolated space on your server for Python projects, ensuring that each of your projects can have its own set of dependencies that won’t disrupt any of your other projects.

Setting up a programming environment provides us with greater control over our Python projects and over how different versions of packages are handled. This is especially important when working with third-party packages.

You can set up as many Python programming environments as you want. Each environment is basically a directory or folder on your server that has a few scripts in it to make it act as an environment.

While there are a few ways to achieve a programming environment in Python, we’ll be using the venv module here, which is part of the standard Python 3 library. Let’s install venv by typing:

sudo apt install -y python3-venv

With this installed, we are ready to create environments. Let’s either choose which directory we would like to put our Python programming environments in, or create a new directory with mkdir, as in:

mkdir environments
cd environments

Once you are in the directory where you would like the environments to live, you can create an environment by running the following command:

python3.5 -m venv my_env

Essentially, pyvenv sets up a new directory that contains a few items which we can view with the ls command:

ls my_env

Outputbin include lib lib64 pyvenv.cfg share

Together, these files work to make sure that your projects are isolated from the broader context of your local machine, so that system files and project files don’t mix. This is good practice for version control and to ensure that each of your projects has access to the particular packages that it needs. Python Wheels, a built-package format for Python that can speed up your software production by reducing the number of times you need to compile, will be in the Ubuntu 18.04 share directory.

To use this environment, you need to activate it, which you can achieve by typing the following command that calls the activate script:

source my_env/bin/activate

Your command prompt will now be prefixed with the name of your environment, in this case it is called my_env. Depending on what version of Debian Linux you are running, your prefix may appear somewhat differently, but the name of your environment in parentheses should be the first thing you see on your line:

This prefix lets us know that the environment my_env is currently active, meaning that when we create programs here they will use only this particular environment’s settings and packages.

Note: Within the virtual environment, you can use the command python instead of python3, and pip instead of pip3 if you would prefer. If you use Python 3 on your machine outside of an environment, you will need to use the python3 and pip3 commands exclusively.

After following these steps, your virtual environment is ready to use.

Step 3 — Creating a “Hello, World” Program

Now that we have our virtual environment set up, let’s create a traditional “Hello, World!” program. This will let us test our environment and provides us with the opportunity to become more familiar with Python if we aren’t already.

To do this, we’ll open up a command-line text editor such as nano and create a new file:

nano hello.py

Once the text file opens up in the terminal window we’ll type out our program:

print("Hello, World!")

Exit nano by typing the CTRL and X keys, and when prompted to save the file press y.

Once you exit out of nano and return to your shell, let’s run the program:

python hello.py

The hello.py program that you just created should cause your terminal to produce the following output:

OutputHello, World!

To leave the environment, simply type the command deactivate and you will return to your original directory.

Conclusion

Congratulations! At this point you have a Python 3 programming environment set up on your Debian 9 Linux server and you can now begin a coding project!

If you are using a local machine rather than a server, refer to the tutorial that is relevant to your operating system in our “How To Install and Set Up a Local Programming Environment for Python 3” series.

With your server ready for software development, you can continue to learn more about coding in Python by reading our free How To Code in Python 3 eBook, or consulting our Programming Project tutorials.

Download our free Python eBook!

How To Code in Python eBook in EPUB format

How To Code in Python eBook in PDF format

How to Install and Use Docker on Debian 9

2018-09-04T21:06:54Z

A previous version of this tutorial was written by finid.

Introduction

Docker is an application that simplifies the process of managing application processes in containers. Containers let you run your applications in resource-isolated processes. They're similar to virtual machines, but containers are more portable, more resource-friendly, and more dependent on the host operating system.

For a detailed introduction to the different components of a Docker container, check out The Docker Ecosystem: An Introduction to Common Components.

In this tutorial, you'll install and use Docker Community Edition (CE) on Debian 9. You'll install Docker itself, work with containers and images, and push an image to a Docker Repository.

Prerequisites

To follow this tutorial, you will need the following:

One Debian 9 server set up by following the Debian 9 initial server setup guide, including a sudo non-root user and a firewall.
An account on Docker Hub if you wish to create your own images and push them to Docker Hub, as shown in Steps 7 and 8.

Step 1 — Installing Docker

The Docker installation package available in the official Debian repository may not be the latest version. To ensure we get the latest version, we'll install Docker from the official Docker repository. To do that, we'll add a new package source, add the GPG key from Docker to ensure the downloads are valid, and then install the package.

First, update your existing list of packages:

sudo apt update

Next, install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common

Then add the GPG key for the official Docker repository to your system:

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

Add the Docker repository to APT sources:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"

Next, update the package database with the Docker packages from the newly added repo:

sudo apt update

Make sure you are about to install from the Docker repo instead of the default Debian repo:

apt-cache policy docker-ce

You'll see output like this, although the version number for Docker may be different:

Output of apt-cache policy docker-ce

docker-ce:
  Installed: (none)
  Candidate: 18.06.1~ce~3-0~debian
  Version table:
     18.06.1~ce~3-0~debian 500
        500 https://download.docker.com/linux/debian stretch/stable amd64 Packages

Notice that docker-ce is not installed, but the candidate for installation is from the Docker repository for Debian 9 (stretch).

Finally, install Docker:

sudo apt install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it's running:

sudo systemctl status docker

The output should be similar to the following, showing that the service is active and running:

Output● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-07-05 15:08:39 UTC; 2min 55s ago
     Docs: https://docs.docker.com
  Main PID: 21319 (dockerd)
   CGroup: /system.slice/docker.service
           ├─21319 /usr/bin/dockerd -H fd://
           └─21326 docker-containerd --config /var/run/docker/containerd/containerd.toml

Installing Docker now gives you not just the Docker service (daemon) but also the docker command line utility, or the Docker client. We'll explore how to use the docker command later in this tutorial.

Step 2 — Executing the Docker Command Without Sudo (Optional)

By default, the docker command can only be run the root user or by a user in the docker group, which is automatically created during Docker's installation process. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you'll get an output like this:

Outputdocker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
See 'docker run --help'.

If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group:

sudo usermod -aG docker ${USER}

To apply the new group membership, log out of the server and back in, or type the following:

su - ${USER}

You will be prompted to enter your user's password to continue.

Confirm that your user is now added to the docker group by typing:

id -nG

Output
sammy sudo docker

If you need to add a user to the docker group that you're not logged in as, declare that username explicitly using:

sudo usermod -aG docker username

The rest of this article assumes you are running the docker command as a user in the docker group. If you choose not to, please prepend the commands with sudo.

Let's explore the docker command next.

Step 3 — Using the Docker Command

Using docker consists of passing it a chain of options and commands followed by arguments. The syntax takes this form:

docker [option] [command] [arguments]

To view all available subcommands, type:

docker

As of Docker 18, the complete list of available subcommands includes:

Output
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

To view the options available to a specific command, type:

docker docker-subcommand --help

To view system-wide information about Docker, use:

docker info

Let's explore some of these commands. We'll start by working with images.

Step 4 — Working with Docker Images

Docker containers are built from Docker images. By default, Docker pulls these images from Docker Hub, a Docker registry managed by Docker, the company behind the Docker project. Anyone can host their Docker images on Docker Hub, so most applications and Linux distributions you'll need will have images hosted there.

To check whether you can access and download images from Docker Hub, type:

docker run hello-world

The output will indicate that Docker in working correctly:

OutputUnable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
9db2ca6ccae0: Pull complete
Digest: sha256:4b8ff392a12ed9ea17784bd3c9a8b1fa3299cac44aca35a85c90c5e3c7afacdc
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

Docker was initially unable to find the hello-world image locally, so it downloaded the image from Docker Hub, which is the default repository. Once the image downloaded, Docker created a container from the image and the application within the container executed, displaying the message.

You can search for images available on Docker Hub by using the docker command with the search subcommand. For example, to search for the Ubuntu image, type:

docker search ubuntu

The script will crawl Docker Hub and return a listing of all images whose name match the search string. In this case, the output will be similar to this:

OutputNAME                                                   DESCRIPTION                                     STARS               OFFICIAL            AUTOMATED
ubuntu                                                 Ubuntu is a Debian-based Linux operating sys…   8320                [OK]
dorowu/ubuntu-desktop-lxde-vnc                         Ubuntu with openssh-server and NoVNC            214                                     [OK]
rastasheep/ubuntu-sshd                                 Dockerized SSH service, built on top of offi…   170                                     [OK]
consol/ubuntu-xfce-vnc                                 Ubuntu container with "headless" VNC session…   128                                     [OK]
ansible/ubuntu14.04-ansible                            Ubuntu 14.04 LTS with ansible                   95                                      [OK]
ubuntu-upstart                                         Upstart is an event-based replacement for th…   88                  [OK]
neurodebian                                            NeuroDebian provides neuroscience research s…   53                  [OK]
1and1internet/ubuntu-16-nginx-php-phpmyadmin-mysql-5   ubuntu-16-nginx-php-phpmyadmin-mysql-5          43                                      [OK]
ubuntu-debootstrap                                     debootstrap --variant=minbase --components=m…   39                  [OK]
nuagebec/ubuntu                                        Simple always updated Ubuntu docker images w…   23                                      [OK]
tutum/ubuntu                                           Simple Ubuntu docker images with SSH access     18
i386/ubuntu                                            Ubuntu is a Debian-based Linux operating sys…   13
1and1internet/ubuntu-16-apache-php-7.0                 ubuntu-16-apache-php-7.0                        12                                      [OK]
ppc64le/ubuntu                                         Ubuntu is a Debian-based Linux operating sys…   12
eclipse/ubuntu_jdk8                                    Ubuntu, JDK8, Maven 3, git, curl, nmap, mc, …   6                                       [OK]
darksheer/ubuntu                                       Base Ubuntu Image -- Updated hourly             4                                       [OK]
codenvy/ubuntu_jdk8                                    Ubuntu, JDK8, Maven 3, git, curl, nmap, mc, …   4                                       [OK]
1and1internet/ubuntu-16-nginx-php-5.6-wordpress-4      ubuntu-16-nginx-php-5.6-wordpress-4             3                                       [OK]
pivotaldata/ubuntu                                     A quick freshening-up of the base Ubuntu doc…   2
1and1internet/ubuntu-16-sshd                           ubuntu-16-sshd                                  1                                       [OK]
ossobv/ubuntu                                          Custom ubuntu image from scratch (based on o…   0
smartentry/ubuntu                                      ubuntu with smartentry                          0                                       [OK]
1and1internet/ubuntu-16-healthcheck                    ubuntu-16-healthcheck                           0                                       [OK]
pivotaldata/ubuntu-gpdb-dev                            Ubuntu images for GPDB development              0
paasmule/bosh-tools-ubuntu                             Ubuntu based bosh-cli                           0                                       [OK]
...

In the OFFICIAL column, OK indicates an image built and supported by the company behind the project. Once you've identified the image that you would like to use, you can download it to your computer using the pull subcommand.

Execute the following command to download the official ubuntu image to your computer:

docker pull ubuntu

You'll see the following output:

OutputUsing default tag: latest
latest: Pulling from library/ubuntu
6b98dfc16071: Pull complete
4001a1209541: Pull complete
6319fc68c576: Pull complete
b24603670dc3: Pull complete
97f170c87c6f: Pull complete
Digest: sha256:5f4bdc3467537cbbe563e80db2c3ec95d548a9145d64453b06939c4592d67b6d
Status: Downloaded newer image for ubuntu:latest

After an image has been downloaded, you can then run a container using the downloaded image with the run subcommand. As you saw with the hello-world example, if an image has not been downloaded when docker is executed with the run subcommand, the Docker client will first download the image, then run a container using it.

To see the images that have been downloaded to your computer, type:

docker images

The output should look similar to the following:

OutputREPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              16508e5c265d        13 days ago         84.1MB
hello-world         latest              2cb0d9787c4d        7 weeks ago         1.85kB

As you'll see later in this tutorial, images that you use to run containers can be modified and used to generate new images, which may then be uploaded (pushed is the technical term) to Docker Hub or other Docker registries.

Let's look at how to run containers in more detail.

Step 5 — Running a Docker Container

The hello-world container you ran in the previous step is an example of a container that runs and exits after emitting a test message. Containers can be much more useful than that, and they can be interactive. After all, they are similar to virtual machines, only more resource-friendly.

As an example, let's run a container using the latest image of Ubuntu. The combination of the -i and -t switches gives you interactive shell access into the container:

docker run -it ubuntu

Your command prompt should change to reflect the fact that you're now working inside the container and should take this form:

Outputroot@d9b100f2f636:/#

Note the container id in the command prompt. In this example, it is d9b100f2f636. You'll need that container ID later to identify the container when you want to remove it.

Now you can run any command inside the container. For example, let's update the package database inside the container. You don't need to prefix any command with sudo, because you're operating inside the container as the root user:

apt update

Then install any application in it. Let's install Node.js:

apt install nodejs

This installs Node.js in the container from the official Ubuntu repository. When the installation finishes, verify that Node.js is installed:

node -v

You'll see the version number displayed in your terminal:

Outputv8.10.0

Any changes you make inside the container only apply to that container.

To exit the container, type exit at the prompt.

Let's look at managing the containers on our system next.

Step 6 — Managing Docker Containers

After using Docker for a while, you'll have many active (running) and inactive containers on your computer. To view the active ones, use:

docker ps

You will see output similar to the following:

OutputCONTAINER ID        IMAGE               COMMAND             CREATED

In this tutorial, you started two containers; one from the hello-world image and another from the ubuntu image. Both containers are no longer running, but they still exist on your system.

To view all containers — active and inactive, run docker ps with the -a switch:

docker ps -a

You'll see output similar to this:

d9b100f2f636        ubuntu              "/bin/bash"         About an hour ago   Exited (0) 8 minutes ago                           sharp_volhard
01c950718166        hello-world         "/hello"            About an hour ago   Exited (0) About an hour ago                       festive_williams

To view the latest container you created, pass it the -l switch:

docker ps -l

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
d9b100f2f636        ubuntu              "/bin/bash"         About an hour ago   Exited (0) 10 minutes ago                       sharp_volhard

To start a stopped container, use docker start, followed by the container ID or the container's name. Let's start the Ubuntu-based container with the ID of d9b100f2f636:

docker start d9b100f2f636

The container will start, and you can use docker ps to see its status:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
d9b100f2f636        ubuntu              "/bin/bash"         About an hour ago   Up 8 seconds                            sharp_volhard

To stop a running container, use docker stop, followed by the container ID or name. This time, we'll use the name that Docker assigned the container, which is sharp_volhard:

docker stop sharp_volhard

Once you've decided you no longer need a container anymore, remove it with the docker rm command, again using either the container ID or the name. Use the docker ps -a command to find the container ID or name for the container associated with the hello-world image and remove it.

docker rm festive_williams

You can start a new container and give it a name using the --name switch. You can also use the --rm switch to create a container that removes itself when it's stopped. See the docker run help command for more information on these options and others.

Containers can be turned into images which you can use to build new containers. Let's look at how that works.

Step 7 — Committing Changes in a Container to a Docker Image

When you start up a Docker image, you can create, modify, and delete files just like you can with a virtual machine. The changes that you make will only apply to that container. You can start and stop it, but once you destroy it with the docker rm command, the changes will be lost for good.

This section shows you how to save the state of a container as a new Docker image.

After installing Node.js inside the Ubuntu container, you now have a container running off an image, but the container is different from the image you used to create it. But you might want to reuse this Node.js container as the basis for new images later.

Then commit the changes to a new Docker image instance using the following command.

docker commit -m "What you did to the image" -a "Author Name" container_id repository/new_image_name

The -m switch is for the commit message that helps you and others know what changes you made, while -a is used to specify the author. The container_id is the one you noted earlier in the tutorial when you started the interactive Docker session. Unless you created additional repositories on Docker Hub, the repository is usually your Docker Hub username.

For example, for the user sammy, with the container ID of d9b100f2f636, the command would be:

docker commit -m "added Node.js" -a "sammy" d9b100f2f636 sammy/ubuntu-nodejs

When you commit an image, the new image is saved locally on your computer. Later in this tutorial, you'll learn how to push an image to a Docker registry like Docker Hub so others can access it.

Listing the Docker images again will show the new image, as well as the old one that it was derived from:

docker images

You'll see output like this:

OutputREPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
sammy/ubuntu-nodejs   latest              7c1f35226ca6        7 seconds ago       179MB
ubuntu                   latest              113a43faa138        4 weeks ago         81.2MB
hello-world              latest              e38bc07ac18e        2 months ago        1.85kB

In this example, ubuntu-nodejs is the new image, which was derived from the existing ubuntu image from Docker Hub. The size difference reflects the changes that were made. And in this example, the change was that NodeJS was installed. So next time you need to run a container using Ubuntu with NodeJS pre-installed, you can just use the new image.

You can also build Images from a Dockerfile, which lets you automate the installation of software in a new image. However, that's outside the scope of this tutorial.

Now let's share the new image with others so they can create containers from it.

Step 8 — Pushing Docker Images to a Docker Repository

The next logical step after creating a new image from an existing image is to share it with a select few of your friends, the whole world on Docker Hub, or other Docker registry that you have access to. To push an image to Docker Hub or any other Docker registry, you must have an account there.

This section shows you how to push a Docker image to Docker Hub. To learn how to create your own private Docker registry, check out How To Set Up a Private Docker Registry on Ubuntu 14.04.

To push your image, first log into Docker Hub.

docker login -u docker-registry-username

You'll be prompted to authenticate using your Docker Hub password. If you specified the correct password, authentication should succeed.

Note: If your Docker registry username is different from the local username you used to create the image, you will have to tag your image with your registry username. For the example given in the last step, you would type:

docker tag sammy/ubuntu-nodejs docker-registry-username/ubuntu-nodejs

Then you may push your own image using:

docker push docker-registry-username/docker-image-name

To push the ubuntu-nodejs image to the sammy repository, the command would be:

docker push sammy/ubuntu-nodejs

The process may take some time to complete as it uploads the images, but when completed, the output will look like this:

Output
The push refers to a repository [docker.io/sammy/ubuntu-nodejs]
e3fbbfb44187: Pushed
5f70bf18a086: Pushed
a3b5c80a4eba: Pushed
7f18b442972b: Pushed
3ce512daaf78: Pushed
7aae4540b42d: Pushed

...

After pushing an image to a registry, it should be listed on your account's dashboard, like that show in the image below.

If a push attempt results in an error of this sort, then you likely did not log in:

Output
The push refers to a repository [docker.io/sammy/ubuntu-nodejs]
e3fbbfb44187: Preparing
5f70bf18a086: Preparing
a3b5c80a4eba: Preparing
7f18b442972b: Preparing
3ce512daaf78: Preparing
7aae4540b42d: Waiting
unauthorized: authentication required

You can now use docker pull sammy/ubuntu-nodejs to pull the image to a new machine and use it to run a new container.

Conclusion

In this tutorial you installed Docker, worked with images and containers, and pushed a modified image to Docker Hub. Now that you know the basics, explore the other Docker tutorials in the DigitalOcean Community.

How To Set Up Time Synchronization on Debian 9

2018-09-04T20:33:23Z

Introduction

Accurate timekeeping has become a critical component of modern software deployments. Whether it's making sure logs are recorded in the right order or database updates are applied correctly, out-of-sync time can cause errors, data corruption, and other hard to debug issues.

Debian 9 has time synchronization built in and activated by default using the standard ntpd time server, provided by the ntp package. In this article we will look at some basic time-related commands, verify that ntpd is active and connected to peers, and learn how to activate the alternate systemd-timesyncd network time service.

Prerequisites

Before starting this tutorial, you will need a Debian 9 server with a non-root, sudo-enabled user, as described in this Debian 9 server setup tutorial.

Navigating Basic Time Commands

The most basic command for finding out the time on your server is date. Any user can type this command to print out the date and time:

date

Output
Tue Sep  4 17:51:49 UTC 2018

Most often your server will default to the UTC time zone, as highlighted in the above output. UTC is Coordinated Universal Time, the time at zero degrees longitude. Consistently using Universal Time reduces confusion when your infrastructure spans multiple time zones.

If you have different requirements and need to change the time zone, you can use the timedatectl command to do so.

First, list the available time zones:

timedatectl list-timezones

A list of time zones will print to your screen. You can press SPACE to page down, and b to page up. Once you find the correct time zone, make note of it then type q to exit the list.

Now set the time zone with timedatectl set-timezone, making sure to replace the highlighted portion below with the time zone you found in the list. You'll need to use sudo with timedatectl to make this change:

sudo timedatectl set-timezone America/New_York

You can verify your changes by running date again:

date

Output
Tue Sep  4 13:52:57 EDT 2018

The time zone abbreviation should reflect the newly chosen value.

Now that we know how to check the clock and set time zones, let’s make sure our time is being synchronized properly.

Checking the Status of ntpd

By default, Debian 9 runs the standard ntpd server to keep your system time synchronized with a pool of external time servers. We can check that it's running with the systemctl command:

sudo systemctl status ntp

Output● ntp.service - LSB: Start NTP daemon
   Loaded: loaded (/etc/init.d/ntp; generated; vendor preset: enabled)
   Active: active (running) since Tue 2018-09-04 15:07:03 EDT; 30min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 876 ExecStart=/etc/init.d/ntp start (code=exited, status=0/SUCCESS)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/ntp.service
           └─904 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:109
. . .

The active (running) status indicates that ntpd started up properly. To get more information about the status of ntpd we can use the ntpq command:

ntpq -p

Output     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
-eterna.binary.n 204.9.54.119     2 u  240  256  377   35.392    0.142   0.211
-static-96-244-9 192.168.10.254   2 u   60  256  377   10.242    1.297   2.412
+minime.fdf.net  83.157.230.212   3 u   99  256  377   24.042    0.128   0.250
*t1.time.bf1.yah 98.139.133.62    2 u   31  256  377   11.112    0.621   0.186
+x.ns.gin.ntt.ne 249.224.99.213   2 u  108  256  377    1.290   -0.073   0.132
-ord1.m-d.net    142.66.101.13    2 u  473  512  377   19.930   -1.764   0.293

ntpq is a query tool for ntpd. The -p flag asks for information about the NTP servers (or peers) ntpd is connected to. Your output will be slightly different, but should list the default Debian pool servers plus a few others. Bear in mind that it can take a few minutes for ntpd to establish connections.

Switching to systemd-timesyncd

It is possible to use systemd's built-in timesyncd component to replace ntpd. timesyncd is a lighter-weight alternative to ntpd that is more integrated with systemd. Note however that it doesn't support running as a time server, and it is slightly less sophisticated in the techniques it uses to keep your system time in sync. If you are running complex real-time distributed systems, you may want to stick with ntpd.

To use timesyncd, we must first uninstall ntpd:

sudo apt purge ntp

Then, start up the timesyncd service:

sudo systemctl start systemd-timesyncd

Finally, check the status of the service to make sure it's running:

sudo systemctl status systemd-timesyncd

Output● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Tue 2018-09-04 16:14:23 EDT; 1s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 3399 (systemd-timesyn)
   Status: "Synchronized to time server 198.60.22.240:123 (0.debian.pool.ntp.org)."
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/systemd-timesyncd.service
           └─3399 /lib/systemd/systemd-timesyncd

We can use timedatectl to print out systemd's current understanding of the time:

timedatectl

Output      Local time: Tue 2018-09-04 16:15:34 EDT
  Universal time: Tue 2018-09-04 20:15:34 UTC
        RTC time: Tue 2018-09-04 20:15:33
       Time zone: America/New_York (EDT, -0400)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

This prints out the local time, universal time (which may be the same as local time, if you didn't switch from the UTC time zone), and some network time status information. Network time on: yes means that timesyncd is enabled, and NTP synchronized: yes indicates that the time has been successfully synced.

Conclusion

In this article we’ve shown how to view the system time, change time zones, work with ntpd, and switch to systemd's timesyncd service. If you have more sophisticated timekeeping needs than what we’ve covered here, you might refer to the offical NTP documentation, and also take a look at the NTP Pool Project, a global group of volunteers providing much of the world's NTP infrastructure.

How To Install Nginx on Debian 9

2018-09-04T19:26:03Z

Introduction

Nginx is one of the most popular web servers in the world and responsible for hosting some of the largest and highest-traffic sites on the internet. It is more resource-friendly than Apache in most cases and can be used as a web server or reverse proxy.

In this guide, we'll discuss how to install Nginx on your Debian 9 server.

Prerequisites

Before you begin this guide, you should have a regular, non-root user with sudo privileges configured on your server and an active firewall. You can learn how to set these up by following our initial server setup guide for Debian 9.

When you have an account available, log in as your non-root user to begin.

Step 1 – Installing Nginx

Because Nginx is available in Debian's default repositories, it is possible to install it from these repositories using the apt packaging system.

Since this is our first interaction with the apt packaging system in this session, let's also update our local package index so that we have access to the most recent package listings. Afterwards, we can install nginx:

sudo apt update
sudo apt install nginx

After accepting the procedure, apt will install Nginx and any required dependencies to your server.

Step 2 – Adjusting the Firewall

Before testing Nginx, the firewall software needs to be adjusted to allow access to the service.

List the application configurations that ufw knows how to work with by typing:

sudo ufw app list

You should get a listing of the application profiles:

OutputAvailable applications:
...
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
...

As you can see, there are three profiles available for Nginx:

Nginx Full: This profile opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
Nginx HTTP: This profile opens only port 80 (normal, unencrypted web traffic)
Nginx HTTPS: This profile opens only port 443 (TLS/SSL encrypted traffic)

You can enable this by typing:

sudo ufw allow 'Nginx HTTP'

You can verify the change by typing:

sudo ufw status

You should see HTTP traffic allowed in the displayed output:

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Nginx HTTP                 ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

Step 3 – Checking your Web Server

At the end of the installation process, Debian 9 starts Nginx. The web server should already be up and running.

We can check with the systemd init system to make sure the service is running by typing:

systemctl status nginx

Output● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-09-04 18:15:57 UTC; 3min 28s ago
     Docs: man:nginx(8)
  Process: 2402 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 2399 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 2404 (nginx)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─2404 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─2405 nginx: worker process

As you can see above, the service appears to have started successfully. However, the best way to test this is to actually request a page from Nginx.

You can access the default Nginx landing page to confirm that the software is running properly by navigating to your server's IP address. If you do not know your server's IP address, try typing this at your server's command prompt:

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'

You will get back a few lines. You can try each in your web browser to see if they work.

When you have your server's IP address, enter it into your browser's address bar:

http://your_server_ip

You should see the default Nginx landing page:

This page is included with Nginx to show you that the server is running correctly.

Step 4 – Managing the Nginx Process

Now that you have your web server up and running, let's review some basic management commands.

To stop your web server, type:

sudo systemctl stop nginx

To start the web server when it is stopped, type:

sudo systemctl start nginx

To stop and then start the service again, type:

sudo systemctl restart nginx

If you are simply making configuration changes, Nginx can often reload without dropping connections. To do this, type:

sudo systemctl reload nginx

By default, Nginx is configured to start automatically when the server boots. If this is not what you want, you can disable this behavior by typing:

sudo systemctl disable nginx

To re-enable the service to start up at boot, you can type:

sudo systemctl enable nginx

Step 5 – Setting Up Server Blocks

When using the Nginx web server, server blocks (similar to virtual hosts in Apache) can be used to encapsulate configuration details and host more than one domain from a single server. We will set up a domain called example.com, but you should replace this with your own domain name. To learn more about setting up a domain name with DigitalOcean, see our introduction to DigitalOcean DNS.

Nginx on Debian 9 has one server block enabled by default that is configured to serve documents out of a directory at /var/www/html. While this works well for a single site, it can become unwieldy if you are hosting multiple sites. Instead of modifying /var/www/html, let's create a directory structure within /var/www for our example.com site, leaving /var/www/html in place as the default directory to be served if a client request doesn't match any other sites.

Create the directory for example.com as follows, using the -p flag to create any necessary parent directories:

sudo mkdir -p /var/www/example.com/html

Next, assign ownership of the directory with the $USER environment variable:

sudo chown -R $USER:$USER /var/www/example.com/html

The permissions of your web roots should be correct if you haven't modified your umask value, but you can make sure by typing:

sudo chmod -R 755 /var/www/example.com

Next, create a sample index.html page using nano or your favorite editor:

nano /var/www/example.com/html/index.html

Inside, add the following sample HTML:

/var/www/example.com/html/index.html

<html>
    <head>
        <title>Welcome to Example.com!</title>
    </head>
    <body>
        <h1>Success!  The example.com server block is working!</h1>
    </body>
</html>

Save and close the file when you are finished.

In order for Nginx to serve this content, it's necessary to create a server block with the correct directives. Instead of modifying the default configuration file directly, let’s make a new one at /etc/nginx/sites-available/example.com:

sudo nano /etc/nginx/sites-available/example.com

Paste in the following configuration block, which is similar to the default, but updated for our new directory and domain name:

/etc/nginx/sites-available/example.com

server {
        listen 80;
        listen [::]:80;

        root /var/www/example.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name example.com www.example.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

Notice that we’ve updated the root configuration to our new directory, and the server_name to our domain name.

Next, let's enable the file by creating a link from it to the sites-enabled directory, which Nginx reads from during startup:

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

Two server blocks are now enabled and configured to respond to requests based on their listen and server_name directives (you can read more about how Nginx processes these directives here):

example.com: Will respond to requests for example.com and www.example.com.
default: Will respond to any requests on port 80 that do not match the other two blocks.

To avoid a possible hash bucket memory problem that can arise from adding additional server names, it is necessary to adjust a single value in the /etc/nginx/nginx.conf file. Open the file:

sudo nano /etc/nginx/nginx.conf

Find the server_names_hash_bucket_size directive and remove the # symbol to uncomment the line:

/etc/nginx/nginx.conf

...
http {
    ...
    server_names_hash_bucket_size 64;
    ...
}
...

Save and close the file when you are finished.

Next, test to make sure that there are no syntax errors in any of your Nginx files:

sudo nginx -t

If there aren't any problems, you will see the following output:

Outputnginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Once your configuration test passes, restart Nginx to enable your changes:

sudo systemctl restart nginx

Nginx should now be serving your domain name. You can test this by navigating to http://example.com, where you should see something like this:

Step 6 – Getting Familiar with Important Nginx Files and Directories

Now that you know how to manage the Nginx service itself, you should take a few minutes to familiarize yourself with a few important directories and files.

Content

/var/www/html: The actual web content, which by default only consists of the default Nginx page you saw earlier, is served out of the /var/www/html directory. This can be changed by altering Nginx configuration files.

Server Configuration

/etc/nginx: The Nginx configuration directory. All of the Nginx configuration files reside here.
/etc/nginx/nginx.conf: The main Nginx configuration file. This can be modified to make changes to the Nginx global configuration.
/etc/nginx/sites-available/: The directory where per-site server blocks can be stored. Nginx will not use the configuration files found in this directory unless they are linked to the sites-enabled directory. Typically, all server block configuration is done in this directory, and then enabled by linking to the other directory.
/etc/nginx/sites-enabled/: The directory where enabled per-site server blocks are stored. Typically, these are created by linking to configuration files found in the sites-available directory.
/etc/nginx/snippets: This directory contains configuration fragments that can be included elsewhere in the Nginx configuration. Potentially repeatable configuration segments are good candidates for refactoring into snippets.

Server Logs

/var/log/nginx/access.log: Every request to your web server is recorded in this log file unless Nginx is configured to do otherwise.
/var/log/nginx/error.log: Any Nginx errors will be recorded in this log.

Conclusion

Now that you have your web server installed, you have many options for the type of content you can serve and the technologies you can use to create a richer experience for your users.

How To Install MariaDB on Debian 9

2018-09-04T19:32:20Z

Introduction

MariaDB is an open-source database management system, commonly installed in place of MySQL as part of the popular LAMP (Linux, Apache, MySQL, PHP/Python/Perl) stack. It uses a relational database and SQL (Structured Query Language) to manage its data. MariaDB was forked from MySQL in 2009 due to licensing concerns.

The short version of the installation is simple: update your package index, install the mariadb-server package (which points to MariaDB), and then run the included security script.

sudo apt update
sudo apt install mariadb-server
sudo mysql_secure_installation

This tutorial will explain how to install MariaDB version 10.1 on a Debian 9 server.

Prerequisites

To follow this tutorial, you will need:

One Debian 9 server set up by following this initial server setup guide, including a non-root user with sudo privileges and a firewall.

Step 1 — Installing MariaDB

On Debian 9, MariaDB version 10.1 is included in the APT package repositories by default. It is marked as the default MySQL variant by the Debian MySQL/MariaDB packaging team.

To install it, update the package index on your server with apt:

sudo apt update

Then install the package:

sudo apt install mariadb-server

This will install MariaDB, but will not prompt you to set a password or make any other configuration changes. Because this leaves your installation of MariaDB insecure, we will address this next.

Step 2 — Configuring MariaDB

For fresh installations, you'll want to run the included security script. This changes some of the less secure default options for things like remote root logins and sample users.

Run the security script:

sudo mysql_secure_installation

The next prompt asks you whether you'd like to set up a database root password. Type N and then press ENTER. In Debian, the root account for MariaDB is tied closely to automated system maintenance, so we should not change the configured authentication methods for that account. Doing so would make it possible for a package update to break the database system by removing access to the administrative account. Later, we will cover how to optionally set up an additional administrative account for password access if socket authentication is not appropriate for your use case.

From there, you can press Y and then ENTER to accept the defaults for all the subsequent questions. This will remove some anonymous users and the test database, disable remote root logins, and load these new rules so that MariaDB immediately respects the changes you have made.

Step 3 — (Optional) Adjusting User Authentication and Privileges

In Debian systems running MariaDB 10.1, the root MariaDB user is set to authenticate using the unix_socket plugin by default rather than with a password. This allows for some greater security and usability in many cases, but it can also complicate things when you need to allow an external program (e.g., phpMyAdmin) administrative rights.

Because the server uses the root account for tasks like log rotation and starting and stopping the server, it is best not to change the root account's authentication details. Changing the account credentials in the /etc/mysql/debian.cnf may work initially, but package updates could potentially overwrite those changes. Instead of modifying the root account, the package maintainers recommend creating a separate administrative account if you need to set up password-based access.

To do so, we will be creating a new account called admin with the same capabilities as the root account, but configured for password authentication. To do this, open up the MariaDB prompt from your terminal:

sudo mysql

Now, we can create a new user with root privileges and password-based access. Change the username and password to match your preferences:

GRANT ALL ON *.* TO 'admin'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;

Flush the privileges to ensure that they are saved and available in the current session:

FLUSH PRIVILEGES;

Following this, exit the MariaDB shell:

exit

Finally, let's test the MariaDB installation.

Step 4 — Testing MariaDB

When installed from the default repositories, MariaDB should start running automatically. To test this, check its status.

sudo systemctl status mariadb

You'll see output similar to the following:

Output

● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-09-04 16:22:47 UTC; 2h 35min ago
  Process: 15596 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSIT
  Process: 15594 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 15478 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   
  Process: 15474 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITI
  Process: 15471 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysql
 Main PID: 15567 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 27 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─15567 /usr/sbin/mysqld

Sep 04 16:22:45 deb-mysql1 systemd[1]: Starting MariaDB database server...
Sep 04 16:22:46 deb-mysql1 mysqld[15567]: 2018-09-04 16:22:46 140183374869056 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-0+deb9u1) starting as process 15567 ...
Sep 04 16:22:47 deb-mysql1 systemd[1]: Started MariaDB database server.

If MariaDB isn't running, you can start it with sudo systemctl start mariadb.

For an additional check, you can try connecting to the database using the mysqladmin tool, which is a client that lets you run administrative commands. For example, this command says to connect to MariaDB as root and return the version using the Unix socket:

sudo mysqladmin version

You should see output similar to this:

Outputmysqladmin  Ver 9.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu on x86_64
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Server version      10.1.26-MariaDB-0+deb9u1
Protocol version    10
Connection      Localhost via UNIX socket
UNIX socket     /var/run/mysqld/mysqld.sock
Uptime:         2 hours 44 min 46 sec

Threads: 1  Questions: 36  Slow queries: 0  Opens: 21  Flush tables: 1  Open tables: 15  Queries per second avg: 0.003

If you configured a separate administrative user with password authentication, you could perform the same operation by typing:

mysqladmin -u admin -p version

This means MariaDB is up and running and that your user is able to authenticate successfully.

Conclusion

You now have a basic MariaDB setup installed on your server. Here are a few examples of next steps you can take:

How To Install Linux, Apache, MariaDB, PHP (LAMP) stack on Debian 9

2018-09-04T18:49:40Z

Introduction

A "LAMP" stack is a group of open source software that is typically installed together to enable a server to host dynamic websites and web apps. This term is actually an acronym which represents the Linux operating system, with the Apache web server. The site data is stored in a MariaDB database, and dynamic content is processed by PHP.

In this guide, we will install a LAMP stack on a Debian 9 server.

Prerequisites

In order to complete this tutorial, you will need to have a Debian 9 server with a non-root sudo-enabled user account and a basic firewall. This can be configured using our initial server setup guide for Debian 9.

Step 1 — Installing Apache and Updating the Firewall

The Apache web server is among the most popular web servers in the world. It's well-documented and has been in wide use for much of the history of the web, which makes it a great default choice for hosting a website.

Install Apache using Debian's package manager, apt:

sudo apt update
sudo apt install apache2

Since this is a sudo command, these operations are executed with root privileges. It will ask you for your regular user's password to verify your intentions.

Once you've entered your password, apt will tell you which packages it plans to install and how much extra disk space they'll take up. Press Y and hit ENTER to continue, and the installation will proceed.

Next, assuming that you have followed the initial server setup instructions by installing and enabling the UFW firewall, make sure that your firewall allows HTTP and HTTPS traffic.

When installed on Debian 9, UFW comes loaded with app profiles which you can use to tweak your firewall settings. View the full list of application profiles by running:

sudo ufw app list

The WWW profiles are used to manage ports used by web servers:

OutputAvailable applications:
. . .
  WWW
  WWW Cache
  WWW Full
  WWW Secure
. . .

If you inspect the WWW Full profile, it shows that it enables traffic to ports 80 and 443:

sudo ufw app info "WWW Full"

OutputProfile: WWW Full
Title: Web Server (HTTP,HTTPS)
Description: Web Server (HTTP,HTTPS)

Ports:
  80,443/tcp

Allow incoming HTTP and HTTPS traffic for this profile:

sudo ufw allow in “WWW Full”

You can do a spot check right away to verify that everything went as planned by visiting your server's public IP address in your web browser:

http://your_server_ip

You will see the default Debian 9 Apache web page, which is there for informational and testing purposes. It should look something like this:

If you see this page, then your web server is now correctly installed and accessible through your firewall.

If you do not know what your server's public IP address is, there are a number of ways you can find it. Usually, this is the address you use to connect to your server through SSH.

There are a few different ways to do this from the command line. First, you could use the iproute2 tools to get your IP address by typing this:

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'

This will give you two or three lines back. They are all correct addresses, but your computer may only be able to use one of them, so feel free to try each one.

An alternative method is to use the curl utility to contact an outside party to tell you how it sees your server. This is done by asking a specific server what your IP address is:

sudo apt install curl
curl http://icanhazip.com

Regardless of the method you use to get your IP address, type it into your web browser's address bar to view the default Apache page.

Step 2 — Installing MariaDB

Now that you have your web server up and running, it is time to install MariaDB. MariaDB is a database management system. Basically, it will organize and provide access to databases where your site can store information.

MariaDB is a community-built fork of MySQL. In Debian 9, the default MySQL server is MariaDB 10.1, and the mysql-server package, which is normally used to install MySQL, is a transitional package that will actually install MariaDB. However, it’s recommended that you install MariaDB using the program’s actual package, mariadb-server.

Again, use apt to acquire and install this software:

sudo apt install mariadb-server

Note: In this case, you do not have to run sudo apt update prior to the command. This is because you recently ran it in the commands above to install Apache, and the package index on your computer should already be up-to-date.

This command, too, will show you a list of the packages that will be installed, along with the amount of disk space they'll take up. Enter Y to continue.

When the installation is complete, run a simple security script that comes pre-installed with MariaDB which will remove some insecure default settings and lock down access to your database system. Start the interactive script by running:

sudo mysql_secure_installation

This will take you through a series of prompts where you can make some changes to your MariaDB installation’s security options. The first prompt will ask you to enter the current database root password. This is an administrative account in MariaDB that has increased privileges. Think of it as being similar to the root account for the server itself (although the one you are configuring now is a MariaDB-specific account). Because you just installed MariaDB and haven’t made any configuration changes yet, this password will be blank, so just press ENTER at the prompt.

sudo mariadb

Now, we can create a new user with root privileges and password-based access. Change the username and password to match your preferences:

GRANT ALL ON *.* TO 'admin'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;

Flush the privileges to ensure that they are saved and available in the current session:

FLUSH PRIVILEGES;

Following this, exit the MariaDB shell:

exit

Now, any time you want to access your database as your new administrative user, you’ll need to authenticate as that user with the password you just set using the following command:

mariadb -u admin -p

At this point, your database system is set up and you can move on to installing PHP, the final component of the LAMP stack.

Step 3 — Installing PHP

PHP is the component of your setup that will process code to display dynamic content. It can run scripts, connect to your MariaDB databases to get information, and hand the processed content over to your web server to display.

Once again, leverage the apt system to install PHP. In addition, include some helper packages this time so that PHP code can run under the Apache server and talk to your MariaDB database:

sudo apt install php libapache2-mod-php php-mysql

This should install PHP without any problems. We'll test this in a moment.

In most cases, you will want to modify the way that Apache serves files when a directory is requested. Currently, if a user requests a directory from the server, Apache will first look for a file called index.html. We want to tell the web server to prefer PHP files over others, so make Apache look for an index.php file first.

To do this, type this command to open the dir.conf file in a text editor with root privileges:

sudo nano /etc/apache2/mods-enabled/dir.conf

It will look like this:

/etc/apache2/mods-enabled/dir.conf

<IfModule mod_dir.c>
    DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
</IfModule>

Move the PHP index file (highlighted above) to the first position after the DirectoryIndex specification, like this:

/etc/apache2/mods-enabled/dir.conf

<IfModule mod_dir.c>
    DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm
</IfModule>

When you are finished, save and close the file by pressing CTRL+X. Confirm the save by typing Y and then hit ENTER to verify the file save location.

After this, restart the Apache web server in order for your changes to be recognized. Do this by typing this:

sudo systemctl restart apache2

You can also check on the status of the apache2 service using systemctl:

sudo systemctl status apache2

Sample Output● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-09-04 18:23:03 UTC; 9s ago
  Process: 22209 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 22216 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 22221 (apache2)
    Tasks: 6 (limit: 4915)
   CGroup: /system.slice/apache2.service
           ├─22221 /usr/sbin/apache2 -k start
           ├─22222 /usr/sbin/apache2 -k start
           ├─22223 /usr/sbin/apache2 -k start
           ├─22224 /usr/sbin/apache2 -k start
           ├─22225 /usr/sbin/apache2 -k start
           └─22226 /usr/sbin/apache2 -k start

To enhance the functionality of PHP, you have the option to install some additional modules. To see the available options for PHP modules and libraries, pipe the results of apt search into less, a pager which lets you scroll through the output of other commands:

apt search php- | less

Use the arrow keys to scroll up and down, and press Q to quit.

The results are all optional components that you can install. It will give you a short description for each:

OutputSorting...
Full Text Search...
bandwidthd-pgsql/stable 2.0.1+cvs20090917-10 amd64
  Tracks usage of TCP/IP and builds html files with graphs

bluefish/stable 2.2.9-1+b1 amd64
  advanced Gtk+ text editor for web and software development

cacti/stable 0.8.8h+ds1-10 all
  web interface for graphing of monitoring systems

cakephp-scripts/stable 2.8.5-1 all
  rapid application development framework for PHP (scripts)

ganglia-webfrontend/stable 3.6.1-3 all
  cluster monitoring toolkit - web front-end

haserl/stable 0.9.35-2+b1 amd64
  CGI scripting program for embedded environments

kdevelop-php-docs/stable 5.0.3-1 all
  transitional package for kdevelop-php

kdevelop-php-docs-l10n/stable 5.0.3-1 all
  transitional package for kdevelop-php-l10n
…
:

To learn more about what each module does, you could search the internet for more information about them. Alternatively, look at the long description of the package by typing:

apt show package_name

There will be a lot of output, with one field called Description which will have a longer explanation of the functionality that the module provides.

For example, to find out what the php-cli module does, you could type this:

apt show php-cli

Along with a large amount of other information, you'll find something that looks like this:

Output…
Description: command-line interpreter for the PHP scripting language (default)
 This package provides the /usr/bin/php command interpreter, useful for
 testing PHP scripts from a shell or performing general shell scripting tasks.
 .
 PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used
 open source general-purpose scripting language that is especially suited
 for web development and can be embedded into HTML.
 .
 This package is a dependency package, which depends on Debian's default
 PHP version (currently 7.0).
…

If, after researching, you decide you would like to install a package, you can do so by using the apt install command like you have been doing for the other software.

If you decided that php-cli is something that you need, you could type:

sudo apt install php-cli

If you want to install more than one module, you can do that by listing each one, separated by a space, following the apt install command, like this:

sudo apt install package1 package2 ...

At this point, your LAMP stack is installed and configured. Before making any more changes or deploying an application, though, it would be helpful to proactively test out your PHP configuration in case there are any issues that should be addressed.

Step 4 — Testing PHP Processing on your Web Server

In order to test that your system is configured properly for PHP, create a very basic PHP script called info.php. In order for Apache to find this file and serve it correctly, it must be saved to a very specific directory called the web root.

In Debian 9, this directory is located at /var/www/html/. Create the file at that location by running:

sudo nano /var/www/html/info.php

This will open a blank file. Add the following text, which is valid PHP code, inside the file:

/var/www/html/info.php

<?php
phpinfo();
?>

When you are finished, save and close the file.

Now you can test whether your web server is able to correctly display content generated by this PHP script. To try this out, visit this page in your web browser. You'll need your server's public IP address again.

The address you will want to visit is:

http://your_server_ip/info.php

The page that you come to should look something like this:

This page provides some basic information about your server from the perspective of PHP. It is useful for debugging and to ensure that your settings are being applied correctly.

If you can see this page in your browser, then your PHP is working as expected.

You probably want to remove this file after this test because it could actually give information about your server to unauthorized users. To do this, run the following command:

sudo rm /var/www/html/info.php

You can always recreate this page if you need to access the information again later.

Conclusion

Now that you have a LAMP stack installed, you have many choices for what to do next. Basically, you've installed a platform that will allow you to install most kinds of websites and web software on your server.

How To Reset Your MySQL or MariaDB Root Password on Ubuntu 18.04

2018-08-31T20:23:48Z

Introduction

Forgetting passwords happens to the best of us. If you forget or lose the root password to your MySQL or MariaDB database, you can still gain access and reset the password if you have access to the server and a user account with sudo privileges.

Note: On fresh Ubuntu 18.04 installations, the default MySQL or MariaDB configuration usually allows you to access the database (with full administrative privileges) without providing a password as long as you make the connection from the system's root account. In this scenario it may not be necessary to reset the password. Before you proceed with resetting your database root password, try to access the database with the sudo mysql command. If this results in an access denied error, follow the steps in this tutorial.

This tutorial demonstrates how to reset the root password for MySQL and MariaDB databases installed with the apt package manager on Ubuntu 18.04. The procedure for changing the root password differs depending on whether you have MySQL or MariaDB installed, as well as the default systemd configuration that ships with the distribution or packages from other vendors. While the instructions in this tutorial may work with other system or database server versions, they have been tested specifically with Ubuntu 18.04 and distribution-supplied packages.

Prerequisites

To recover your MySQL or MariaDB root password, you will need:

Access to the Ubuntu 18.04 server running MySQL or MariaDB with a sudo user or other way of accessing the server with root privileges. In order to try out the recovery methods in this tutorial without affecting your production server, use the initial server setup tutorial to create a test server with a regular, non-root user with sudo privileges. Then install MySQL following How to install MySQL on Ubuntu 18.04.

Step 1 — Identifying the Database Version and Stopping the Server

Ubuntu 18.04 runs either MySQL or MariaDB, a popular drop-in replacement which is fully compatible with MySQL. You'll need to use different commands to recover the root password depending on which of these you have installed, so follow the steps in this section to determine which database server you're running.

Check your version with the following command:

mysql --version

If you're running MariaDB, you'll see "MariaDB" preceded by the version number in the output:

MariaDB output
mysql  Ver 15.1 Distrib 10.1.29-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

You'll see output like this if you're running MySQL:

MySQL output
mysql  Ver 14.14 Distrib 5.7.22, for Linux (x86_64) using  EditLine wrapper

Make note of which database, as this determines the appropriate commands to follow in the rest of this tutorial.

In order to change the root password, you'll need to shut down the database server. If you're running MariaDB, you can do so with the following command:

sudo systemctl stop mariadb

For MySQL, shut down the database server by running:

sudo systemctl stop mysql

With the database stopped, you can restart it in safe mode to reset the root password.

Step 2 — Restarting the Database Server Without Permission Checks

Running MySQL and MariaDB without permission checking allows accessing the database command line with root privileges without providing a valid password. To do this, you need to stop the database from loading the grant tables, which store user privilege information. Since this is a bit of a security risk, you may also want to disable networking to prevent other clients from connecting to the temporarily vulnerable server.

Depending on which database server you've installed, the way of starting the server without loading the grant tables differs.

Configuring MariaDB to Start Without Grant Tables

In order to start the MariaDB server without the grant tables, we'll use the systemd unit file to set additional parameters for the MariaDB server daemon.

Execute the following command which sets the MYSQLD_OPTS environment variable used by MariaDB upon startup. The --skip-grant-tables and --skip-networking options tell MariaDB to start up without loading the grant tables or networking features:

sudo systemctl set-environment MYSQLD_OPTS="--skip-grant-tables --skip-networking"

Then start the MariaDB server:

sudo systemctl start mariadb

This command won't produce any output, but it will restart the database server, taking into account the new environment variable settings.

You can ensure it started with sudo systemctl status mariadb.

Now you should be able to connect to the database as the MariaDB root user without supplying a password:

sudo mysql -u root

You'll immediately see a database shell prompt:

MariaDB promptType 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

Now that you have access to the database server, you can change the root password as shown in Step 3.

Configuring MySQL to Start Without Grant Tables

In order to start the MySQL server without its grant tables, you'll alter the systemd configuration for MySQL to pass additional command-line parameters to the server upon startup.

To do this, execute the following command:

sudo systemctl edit mysql

This command will open a new file in the nano editor, which you'll use to edit MySQL's service overrides. These change the default service parameters for MySQL. This file will be empty, so add the following content:

MySQL service overrides

[Service]
ExecStart=
ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid --skip-grant-tables --skip-networking

The first ExecStart statement clears the default value, while the second one provides systemd with the new startup command including parameters to disable loading the grant tables and networking capabilities.

Press CTRL-x to exit the file, then Y to save the changes that you made, then ENTER to confirm the file name.

Reload the systemd configuration to apply these changes:

sudo systemctl daemon-reload

Now start the MySQL server:

sudo systemctl start mysql

The command will show no output, but the database server will start. The grant tables and networking will not be enabled.

Connect to the database as the root user:

sudo mysql -u root

You'll immediately see a database shell prompt:

MySQL promptType 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Now that you have access to the server, you can change the root password.

Step 3 — Changing the Root Password

The database server is now running in a limited mode; the grant tables are not loaded and there's no networking support enabled. This lets you access the server without providing a password, but it prohibits you from executing commands that alter data. To reset the root password, you must load the grant tables now that you've gained access to the server.

Tell the database server to reload the grant tables by issuing the FLUSH PRIVILEGES command.

FLUSH PRIVILEGES;

You can now change the root password. The method you use depends on whether you are using MariaDB or MySQL.

Changing the MariaDB Password

If you are using MariaDB, execute the following statement to set the password for the root account, making sure to replace new_password with a strong new password that you'll remember.

UPDATE mysql.user SET password = PASSWORD('new_password') WHERE user = 'root';

You'll see this output indicating that the password changed:

OutputQuery OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB allows using custom authentication mechanisms, so execute the following two statements to make sure MariaDB will use its default authentication mechanism for the new password you assigned to the root account:

UPDATE mysql.user SET authentication_string = '' WHERE user = 'root';
UPDATE mysql.user SET plugin = '' WHERE user = 'root';

You'll see the following output for each statement:

OutputQuery OK, 0 rows affected (0.01 sec)
Rows matched: 1  Changed: 0  Warnings: 0

The password is now changed. Type exit to exit the MariaDB console and proceed to Step 4 to restart the database server in normal mode.

Changing the MySQL Password

For MySQL, execute the following statement to change the root user's password, replacing new_password with a strong password you'll remember:

UPDATE mysql.user SET authentication_string = PASSWORD('new_password') WHERE user = 'root';

You'll see this output indicating the password was changed successfully:

OutputQuery OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MySQL allows using custom authentication mechanisms, so execute the following statement to tell MySQL touse its default authentication mechanism to authenticate the root user using the new password:

UPDATE mysql.user SET plugin = 'mysql_native_password' WHERE user = 'root';

You'll see output similar to the previous command:

OutputQuery OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

The password is now changed. Exit the MySQL console by typing exit.

Let's restart the database in normal operational mode.

Step 4 — Reverting Your Database Server to Normal Settings

In order to restart the database server in its normal mode, you have to revert the changes you made so that networking is enabled and the grant tables are loaded. Again, the method you use depends on whether you used MariaDB or MySQL.

For MariaDB, unset the MYSQLD_OPTS environment variable you set previously:

sudo systemctl unset-environment MYSQLD_OPTS

Then, restart the service using systemctl:

sudo systemctl restart mariadb

For MySQL, remove the modified systemd configuration:

sudo systemctl revert mysql

You'll see output similar to the following:

OutputRemoved /etc/systemd/system/mysql.service.d/override.conf.
Removed /etc/systemd/system/mysql.service.d.

Then, reload the systemd configuration to apply the changes:

sudo systemctl daemon-reload

Finally, restart the service:

sudo systemctl restart mysql

The database is now restarted and is back to its normal state. Confirm that the new password works by logging in as the root user with a password:

mysql -u root -p

You'll be prompted for a password. Enter your new password and you'll gain access to the database prompt as expected.

Conclusion

You have restored administrative access to the MySQL or MariaDB server. Make sure the new password you chose is strong and secure and keep it in a safe place.

For more information on user management, authentication mechanisms, or ways of resetting database password for other version of MySQL or MariaDB, please refer to the official MySQL documentation or MariaDB documentation.

How To Install Node.js on Debian 9

2018-09-04T17:08:58Z

Introduction

Node.js is a JavaScript platform for general-purpose programming that allows users to build network applications quickly. By leveraging JavaScript on both the front and backend, Node.js makes development more consistent and integrated.

In this guide, we'll show you how to get started with Node.js on a Debian 9 server.

Prerequisites

This guide assumes that you are using Debian 9. Before you begin, you should have a non-root user account with sudo privileges set up on your system. You can learn how to set this up by following the initial server setup for Debian 9.

Installing the Distro-Stable Version for Debian

Debian contains a version of Node.js in its default repositories. At the time of writing, this version is 4.8.2, which will reach end-of-life at the end of April 2018. If you would like to experiment with the language using a stable and sufficient option, then installing from the repositories may make sense. It is recommended, however, that for development and production use cases you install a more recent version with a PPA. We will discuss how to install from a PPA in the next step.

To get the distro-stable version of Node.js, you can use the apt package manager. First, refresh your local package index:

sudo apt update

Then install the Node.js package from the repositories:

sudo apt install nodejs

If the package in the repositories suits your needs, then this is all you need to do to get set up with Node.js.

To check which version of Node.js you have installed after these initial steps, type:

nodejs -v

Because of a conflict with another package, the executable from the Debian repositories is called nodejs instead of node. Keep this in mind as you are running software.

Once you have established which version of Node.js you have installed from the Debian repositories, you can decide whether or not you would like to work with different versions, package archives, or version managers. Next, we'll discuss these elements, along with more flexible and robust methods of installation.

Installing Using a PPA

To work with a more recent version of Node.js, you can add the PPA (personal package archive) maintained by NodeSource. This will have more up-to-date versions of Node.js than the official Debian repositories, and will allow you to choose between Node.js v4.x (the older long-term support version, which will be supported until the end of April 2018), Node.js v6.x (supported until April of 2019), Node.js v8.x (the current LTS version, supported until December of 2019), and Node.js v10.x (the latest version, supported until April of 2021).

Let's first update the local package index and install curl, which you will use to access the PPA:

sudo apt update
sudo apt install curl

Next, let's install the PPA in order to get access to its contents. From your home directory, use curl to retrieve the installation script for your preferred version, making sure to replace 10.x with your preferred version string (if different):

cd ~
curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh

You can inspect the contents of this script with nano or your preferred text editor:

nano nodesource_setup.sh

Run the script under sudo:

sudo bash nodesource_setup.sh

The PPA will be added to your configuration and your local package cache will be updated automatically. After running the setup script, you can install the Node.js package in the same way you did above:

sudo apt install nodejs

To check which version of Node.js you have installed after these initial steps, type:

nodejs -v

Output
v10.9.0

The nodejs package contains the nodejs binary as well as npm, so you don't need to install npm separately.

npm -v

Output
6.2.0

In order for some npm packages to work (those that require compiling code from source, for example), you will need to install the build-essential package:

sudo apt install build-essential

You now have the necessary tools to work with npm packages that require compiling code from source.

Installing Using NVM

An alternative to installing Node.js through apt is to use a tool called nvm, which stands for "Node.js Version Manager". Rather than working at the operating system level, nvm works at the level of an independent directory within your home directory. This means that you can install multiple self-contained versions of Node.js without affecting the entire system.

Controlling your environment with nvm allows you to access the newest versions of Node.js and retain and manage previous releases. It is a different utility from apt, however, and the versions of Node.js that you manage with it are distinct from those you manage with apt.

To download the nvm installation script from the project's GitHub page, you can use curl. Note that the version number may differ from what is highlighted here:

curl -sL https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh -o install_nvm.sh

Inspect the installation script with nano:

nano install_nvm.sh

Run the script with bash:

bash install_nvm.sh

It will install the software into a subdirectory of your home directory at ~/.nvm. It will also add the necessary lines to your ~/.profile file to use the file.

To gain access to the nvm functionality, you'll need to either log out and log back in again or source the ~/.profile file so that your current session knows about the changes:

source ~/.profile

With nvm installed, you can install isolated Node.js versions. For information about the versions of Node.js that are available, type:

nvm ls-remote

Output...
         v8.11.1   (Latest LTS: Carbon)
         v9.0.0
         v9.1.0
         v9.2.0
         v9.2.1
         v9.3.0
         v9.4.0
         v9.5.0
         v9.6.0
         v9.6.1
         v9.7.0
         v9.7.1
         v9.8.0
         v9.9.0
        v9.10.0
        v9.10.1
        v9.11.0
        v9.11.1
        v10.0.0  
        v10.1.0
        v10.2.0
        v10.2.1
        v10.3.0
        v10.4.0
        v10.4.1
        v10.5.0
        v10.6.0
        v10.7.0
        v10.8.0
        v10.9.0

As you can see, the current LTS version at the time of this writing is v8.11.1. You can install that by typing:

nvm install 8.11.1

Usually, nvm will switch to use the most recently installed version. You can tell nvm to use the version you just downloaded by typing:

nvm use 8.11.1

When you install Node.js using nvm, the executable is called node. You can see the version currently being used by the shell by typing:

node -v

Output
v8.11.1

If you have multiple Node.js versions, you can see what is installed by typing:

nvm ls

If you wish to default one of the versions, type:

nvm alias default 8.11.1

This version will be automatically selected when a new session spawns. You can also reference it by the alias like this:

nvm use default

Each version of Node.js will keep track of its own packages and has npm available to manage these.

You can also have npm install packages to the Node.js project's ./node_modules directory. Use the following syntax to install the express module:

npm install express

If you'd like to install the module globally, making it available to other projects using the same version of Node.js, you can add the -g flag:

npm install -g express

This will install the package in:

~/.nvm/versions/node/node_version/lib/node_modules/express

Installing the module globally will let you run commands from the command line, but you'll have to link the package into your local sphere to require it from within a program:

npm link express

You can learn more about the options available to you with nvm by typing:

nvm help

Removing Node.js

You can uninstall Node.js using apt or nvm, depending on the version you want to target. To remove versions installed from the repositories or from the PPA, you will need to work with the apt utility at the system level.

To remove either of these versions, type the following:

sudo apt remove nodejs

This command will remove the package and the configuration files.

To uninstall a version of Node.js that you have enabled using nvm, first determine whether or not the version you would like to remove is the current active version:

nvm current

If the version you are targeting is not the current active version, you can run:

nvm uninstall node_version

This command will uninstall the selected version of Node.js.

If the version you would like to remove is the current active version, you must first deactivate nvm to enable your changes:

nvm deactivate

You can now uninstall the current version using the uninstall command above, which will remove all files associated with the targeted version of Node.js except the cached files that can be used for reinstallation.

Conclusion

There are a quite a few ways to get up and running with Node.js on your Debian 9 server. Your circumstances will dictate which of the above methods is best for your needs. While using the packaged version in the Debian repository is an option for experimentation, installing from a PPA and working with npm or nvm offers additional flexibility.

Initial Server Setup with Debian 9

2018-08-31T19:46:06Z

Introduction

When you first create a new Debian 9 server, there are a few configuration steps that you should take early on as part of the basic setup. This will increase the security and usability of your server and will give you a solid foundation for subsequent actions.

Step One — Logging in as Root

To log into your server, you will need to know your server's public IP address. You will also need the password or, if you installed an SSH key for authentication, the private key for the root user's account. If you have not already logged into your server, you may want to follow our guide on how to connect to your Droplet with SSH, which covers this process in detail.

If you are not already connected to your server, go ahead and log in as the root user using the following command (substitute the highlighted portion of the command with your server's public IP address):

ssh root@your_server_ip

Accept the warning about host authenticity if it appears. If you are using password authentication, provide your root password to log in. If you are using an SSH key that is passphrase protected, you may be prompted to enter the passphrase the first time you use the key each session. If this is your first time logging into the server with a password, you may also be prompted to change the root password.

About Root

The root user is the administrative user in a Linux environment that has very broad privileges. Because of the heightened privileges of the root account, you are discouraged from using it on a regular basis. This is because part of the power inherent with the root account is the ability to make very destructive changes, even by accident.

The next step is to set up an alternative user account with a reduced scope of influence for day-to-day work. We'll teach you how to gain increased privileges during the times when you need them.

Step Two — Creating a New User

Once you are logged in as root, we're prepared to add the new user account that we will use to log in from now on.

sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting

These messages are harmless, but if you wish to avoid them, it is safe to remove the unscd package if you do not not plan on using systems like LDAP for user information:

apt remove unscd

This example creates a new user called sammy, but you should replace it with a username that you like:

adduser sammy

You will be asked a few questions, starting with the account password.

Enter a strong password and, optionally, fill in any of the additional information if you would like. This is not required and you can just hit ENTER in any field you wish to skip.

Step Three — Granting Administrative Privileges

Now, we have a new user account with regular account privileges. However, we may sometimes need to do administrative tasks.

To avoid having to log out of our normal user and log back in as the root account, we can set up what is known as "superuser" or root privileges for our normal account. This will allow our normal user to run commands with administrative privileges by putting the word sudo before each command.

To add these privileges to our new user, we need to add the new user to the sudo group. By default, on Debian 9, users who belong to the sudo group are allowed to use the sudo command.

As root, run this command to add your new user to the sudo group (substitute the highlighted word with your new user):

usermod -aG sudo sammy

Now, when logged in as your regular user, you can type sudo before commands to perform actions with superuser privileges.

Step Four — Setting Up a Basic Firewall

Debian servers can use firewalls to make sure only connections to certain services are allowed. Although the iptables firewall is installed by default, Debian does not strongly recommend any specific firewall. In this guide, we will install and use the UFW firewall to help set policies and manage exceptions.

We can use the apt package manager to install UFW. Update the local index to retrieve the latest information about available packages and then install the firewall by typing:

apt update
apt install ufw

Note: If your servers are running on DigitalOcean, you can optionally use DigitalOcean Cloud Firewalls instead of the UFW firewall. We recommend using only one firewall at a time to avoid conflicting rules that may be difficult to debug.

Firewall profiles allow UFW to manage sets of firewall rules for applications by name. Profiles for some common software are bundled with UFW by default and packages can register additional profiles with UFW during the installation process. OpenSSH, the service allowing us to connect to our server now, has a firewall profile that we can use.

You can see this by typing:

ufw app list

OutputAvailable applications:
  . . .
  OpenSSH
  . . .

We need to make sure that the firewall allows SSH connections so that we can log back in next time. We can allow these connections by typing:

ufw allow OpenSSH

Afterwards, we can enable the firewall by typing:

ufw enable

Type "y" and press ENTER to proceed. You can see that SSH connections are still allowed by typing:

ufw status

OutputStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

As the firewall is currently blocking all connections except for SSH, if you install and configure additional services, you will need to adjust the firewall settings to allow acceptable traffic in. You can learn some common UFW operations in this guide.

Step Five — Enabling External Access for Your Regular User

Now that we have a regular user for daily use, we need to make sure we can SSH into the account directly.

Note: Until verifying that you can log in and use sudo with your new user, we recommend staying logged in as root. This way, if you have problems, you can troubleshoot and make any necessary changes as root. If you are using a DigitalOcean Droplet and experience problems with your root SSH connection, you can log into the Droplet using the DigitalOcean Console.

The process for configuring SSH access for your new user depends on whether your server's root account uses a password or SSH keys for authentication.

If the Root Account Uses Password Authentication

If you logged in to your root account using a password, then password authentication is enabled for SSH. You can SSH to your new user account by opening up a new terminal session and using SSH with your new username:

ssh sammy@your_server_ip

After entering your regular user's password, you will be logged in. Remember, if you need to run a command with administrative privileges, type sudo before it like this:

sudo command_to_run

You will be prompted for your regular user password when using sudo for the first time each session (and periodically afterwards).

To enhance your server's security, we strongly recommend setting up SSH keys instead of using password authentication. Follow our guide on setting up SSH keys on Debian 9 to learn how to configure key-based authentication.

If the Root Account Uses SSH Key Authentication

If you logged in to your root account using SSH keys, then password authentication is disabled for SSH. You will need to add a copy of your local public key to the new user's ~/.ssh/authorized_keys file to log in successfully.

Since your public key is already in the root account's ~/.ssh/authorized_keys file on the server, we can copy that file and directory structure to our new user account in our existing session with the cp command. Afterwards, we can adjust ownership of the files using the chown command.

Make sure to change the highlighted portions of the command below to match your regular user's name:

cp -r ~/.ssh /home/sammy
chown -R sammy:sammy /home/sammy/.ssh

Now, open up a new terminal session and using SSH with your new username:

ssh sammy@your_server_ip

You should be logged in to the new user account without using a password. Remember, if you need to run a command with administrative privileges, type sudo before it like this:

sudo command_to_run

You will be prompted for your regular user password when using sudo for the first time each session (and periodically afterwards).

Step Six — Completing Optional Configuration

Now that we have a strong baseline configuration, we can consider a few optional steps to make the system more accessible. The following sections cover a few additional tweaks focused on usability.

Installing man Pages

Debian provides extensive manuals for most software in the form of man pages. However, the man command is not always included by default on minimal installations.

Install the man-db package to install the man command and the manual databases:

sudo apt install man-db

Now, to view the manual for a component, you can type:

man command

For example, to view the manual for the top command, type:

man top

Most packages in the Debian repositories include manual pages as part of their installation.

Changing the Default Editor

Debian offers a wide variety of text editors, some of which are included in the base system. Commands with integrated editor support, like visudo and systemctl edit, pass text to the editor command, which is mapped to the system default editor. Setting the default editor according to your preferences can help you configure your system more easily and avoid frustration.

If your preferred editor is not installed by default, use apt to install it first:

sudo apt install your_preferred_editor

Next, you can view the current default and modify the selection using the update-alternatives command:

sudo update-alternatives --config editor

The command displays a table of the editors it knows about with a prompt to change the default:

OutputThere are 8 choices for the alternative editor (providing /usr/bin/editor).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /usr/bin/joe         70        auto mode
  1            /bin/nano            40        manual mode
  2            /usr/bin/jmacs       50        manual mode
  3            /usr/bin/joe         70        manual mode
  4            /usr/bin/jpico       50        manual mode
  5            /usr/bin/jstar       50        manual mode
  6            /usr/bin/rjoe        25        manual mode
  7            /usr/bin/vim.basic   30        manual mode
  8            /usr/bin/vim.tiny    15        manual mode

Press <enter> to keep the current choice[*], or type selection number:

The asterisk in the far left column indicates the current selection. To change the default, type the "Selection" number for your preferred editor and press Enter. For example, to use nano as the default editor given the above table, we would choose 1:

Output
Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /bin/nano to provide /usr/bin/editor (editor) in manual mode

From now on, your preferred editor will be used by commands like visudo and systemctl edit, or when the editor command is called.

Where To Go From Here?

At this point, you have a solid foundation for your server. You can install any of the software you need on your server now.

How To Set Up Logical Replication with PostgreSQL 10 on Ubuntu 18.04

2018-08-22T22:02:39Z

Introduction

When setting up an application for production, it's often useful to have multiple copies of your database in place. The process of keeping database copies in sync is called replication. Replication can provide high-availability horizontal scaling for high volumes of simultaneous read operations, along with reduced read latencies. It also allows for peer-to-peer replication between geographically distributed database servers.

PostgreSQL is an open-source object-relational database system that is highly extensible and compliant with ACID (Atomicity, Consistency, Isolation, Durability) and the SQL standard. Version 10.0 of PostgreSQL introduced support for logical replication, in addition to physical replication. In a logical replication scheme, high-level write operations are streamed from a master database server into one or more replica database servers. In a physical replication scheme, binary write operations are instead streamed from master to replica, producing a byte-for-byte exact copy of the original content. In cases where you would like to target a particular subset of data, such as off-load reporting, patching, or upgrading, logical replication can offer speed and flexibility.

In this tutorial, you will configure logical replication with PostgreSQL 10 on two Ubuntu 18.04 servers, with one server acting as the master and the other as the replica. By the end of the tutorial you will be able to replicate data from the master server to the replica using logical replication.

Prerequisites

To follow this tutorial, you will need:

Two Ubuntu 18.04 servers, which we'll name db-master and db-replica, each set up with a regular user account and sudo privileges. To set these up, follow this initial server setup tutorial.
Private networking enabled on your servers. Private networking allows for communication between your servers without the security risks associated with exposing databases to the public internet.
PostgreSQL 10 installed on both servers, following Step 1 of How To Install and Use PostgreSQL on Ubuntu 18.04.

Step 1 — Configuring PostgreSQL for Logical Replication

There are several configuration settings you will need to modify to enable logical replication between your servers. First, you'll configure Postgres to listen on the private network interface instead of the public one, as exposing data over the public network is a security risk. Then you'll configure the appropriate settings to allow replication to db-replica.

On db-master, open /etc/postgresql/10/main/postgresql.conf, the main server configuration file:

sudo nano /etc/postgresql/10/main/postgresql.conf

Find the following line:

/etc/postgresql/10/main/postgresql.conf

...
#listen_addresses = 'localhost'         # what IP address(es) to listen on;
...

Uncomment it by removing the #, and add your db_master_private_ip_address to enable connections on the private network:

Note: In this step and the steps that follow, make sure to use the private IP addresses of your servers, and not their public IPs. Exposing a database server to the public internet is a considerable security risk.

/etc/postgresql/10/main/postgresql.conf

...
listen_addresses = 'localhost, db_master_private_ip_address'
...

This makes db-master listen for incoming connections on the private network in addition to the loopback interface.

Next, find the following line:

/etc/postgresql/10/main/postgresql.conf

...
#wal_level = replica                    # minimal, replica, or logical
...

Uncomment it, and change it to set the PostgreSQL Write Ahead Log (WAL) level to logical. This increases the volume of entries in the log, adding the necessary information for extracting discrepancies or changes to particular data sets:

/etc/postgresql/10/main/postgresql.conf

...
wal_level = logical
...

The entries on this log will be consumed by the replica server, allowing for the replication of the high-level write operations from the master.

Save the file and close it.

Next, let's edit /etc/postgresql/10/main/pg_hba.conf, the file that controls allowed hosts, authentication, and access to databases:

sudo nano /etc/postgresql/10/main/pg_hba.conf

After the last line, let's add a line to allow incoming network connections from db-replica. We'll use db-replica's private IP address, and specify that connections are allowed from all users and databases:

/etc/postgresql/10/main/pg_hba.conf

...
# TYPE      DATABASE        USER            ADDRESS                               METHOD
...
host         all            all             db_replica_private_ip_address/32      md5

Incoming network connections will now be allowed from db-replica, authenticated by a password hash (md5).

Save the file and close it.

Next, let's set our firewall rules to allow traffic from db-replica to port 5432 on db-master:

sudo ufw allow from db_replica_private_ip_address to any port 5432

Finally, restart the PostgreSQL server for the changes to take effect:

sudo systemctl restart postgresql

With your configuration set to allow logical replication, you can now move on to creating a database, user role, and table.

Step 2 — Setting Up a Database, User Role, and Table

To test the functionality of your replication settings, let's create a database, table, and user role. You will create an example database with a sample table, which you can then use to test logical replication between your servers. You will also create a dedicated user and assign them privileges over both the database and the table.

First, open the psql prompt as the postgres user with the following command on both db-master and db-replica:

sudo -u postgres psql

sudo -u postgres psql

Create a new database called example on both hosts:

CREATE DATABASE example;

CREATE DATABASE example;

Note: The final ; in these commands is required. On interactive sessions, PostgreSQL will not execute SQL commands until you terminate them with a semicolon. Meta-commands (those starting with a backslash, like \q and \c) directly control the psql client itself, and are therefore exempt from this rule. For more on meta-commands and the psql client, please refer to the PostgreSQL documentation.

Using the \connect meta-command, connect to the databases you just created on each host:

\c example

\c example

Create a new table called widgets with arbitrary fields on both hosts:

CREATE TABLE widgets
(
    id SERIAL,
    name TEXT,
    price DECIMAL,
    CONSTRAINT widgets_pkey PRIMARY KEY (id)
);

CREATE TABLE widgets
(
    id SERIAL,
    name TEXT,
    price DECIMAL,
    CONSTRAINT widgets_pkey PRIMARY KEY (id)
);

The table on db-replica does not need to be identical to its db-master counterpart. However, it must contain every single column present on the table at db-master. Additional columns must not have NOT NULL or other constraints. If they do, replication will fail.

On db-master, let's create a new user role with the REPLICATION option and a login password. The REPLICATION attribute must be assigned to any role used for replication. We will call our user sammy, but you can replace this with your own username. Make sure to also replace my_password with your own secure password:

CREATE ROLE sammy WITH REPLICATION LOGIN PASSWORD 'my_password';

Make a note of your password, as you will use it later on db-replica to set up replication.

Still on db-master, grant full privileges on the example database to the user role you just created:

GRANT ALL PRIVILEGES ON DATABASE example TO sammy;

Next, grant privileges on all of the tables contained in the database to your user:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO sammy;

The public schema is a default schema in each database into which tables are automatically placed.

With these privileges set, you can now move on to making the tables in your example database available for replication.

Step 3 — Setting Up a Publication

Publications are the mechanism that PostgreSQL uses to make tables available for replication. The database server will keep track internally of the connection and replication status of any replica servers associated with a given publication. On db-master, you will create a publication, my_publication, that will function as a master copy of the data that will be sent to your subscribers — in our case, db-replica.

On db-master, create a publication called my_publication:

CREATE PUBLICATION my_publication;

Add the widgets table you created previously to it:

ALTER PUBLICATION my_publication ADD TABLE widgets;

With your publication in place, you can now add a subscriber that will pull data from it.

Step 4 — Creating a Subscription

Subscriptions are used by PostgreSQL to connect to existing publications. A publication can have many subscriptions across different replica servers, and replica servers can also have their own publications with subscribers. To access the data from the table you created on db-master, you will need to create a subscription to the publication you created in the previous step, my_publication.

On db-replica, let's create a subscription called my_subscription. The CREATE SUBSCRIPTION command will name the subscription, while the CONNECTION parameter will define the connection string to the publisher. This string will include the master server's connection details and login credentials, including the username and password you defined earlier, along with the name of the example database. Once again, remember to use db-master's private IP address, and replace my_password with your own password:

CREATE SUBSCRIPTION my_subscription CONNECTION 'host=db_master_private_ip_address port=5432 password=my_password user=sammy dbname=example' PUBLICATION my_publication;

You will see the following output confirming the subscription:

OutputNOTICE:  created replication slot "my_subscription" on publisher
CREATE SUBSCRIPTION

Upon creating a subscription, PostgreSQL will automatically sync any pre-existing data from the master to the replica. In our case there is no data to sync since the widgets table is empty, but this is a useful feature when adding new subscriptions to an existing database.

With a subscription in place, let's test the setup by adding some demo data to the widgets table.

Step 5 — Testing and Troubleshooting

To test replication between our master and replica, let's add some data to the widgets table and verify that it replicates correctly.

On db-master, insert the following data on the widgets table:

INSERT INTO widgets (name, price) VALUES ('Hammer', 4.50), ('Coffee Mug', 6.20), ('Cupholder', 3.80);

On db-replica, run the following query to fetch all the entries on this table:

SELECT * FROM widgets;

You should now see:

Output id |    name    | price 
----+------------+-------
  1 | Hammer     |  4.50
  2 | Coffee Mug |  6.20
  3 | Cupholder  |  3.80
(3 rows)

Success! The entries have been successfully replicated from db-master to db-replica. From now on, all INSERT, UPDATE, and DELETE queries will be replicated across servers unidirectionally.

One thing to note about write queries on replica servers is that they are not replicated back to the master server. PostgreSQL currently has limited support for resolving conflicts when the data between servers diverges. If there is a conflict, the replication will stop and PostgreSQL will wait until the issue is manually fixed by the database administrator. For that reason, most applications will direct all write operations to the master server, and distribute reads among available replica servers.

You can now exit the psql prompt on both servers:

\q

\q

Now that you have finished testing your setup, you can add and replicate data on your own.

Troubleshooting

If replication doesn't seem to be working, a good first step is checking the PostgreSQL log on db-replica for any possible errors:

tail /var/log/postgresql/postgresql-10-main.log

Here are some common problems that can prevent replication from working:

Private networking is not enabled on both servers, or the servers are on different networks;
db-master is not configured to listen for connections on the correct private network IP;
The Write Ahead Log level on db-master is incorrectly configured (it must be set to logical);
db-master is not configured to accept incoming connections from the correct db-replica private IP address;
A firewall like UFW is blocking incoming PostgreSQL connections on port 5432;
There are mismatched table names or fields between db-master and db-replica;
The sammy database role is missing the required permissions to access the example database on db-master;
The sammy database role is missing the REPLICATION option on db-master;
The sammy database role is missing the required permissions to access the widgets table on db-master;
The table wasn't added to the publication on db-master.

After resolving the existing problem(s), replication should take place automatically. If it doesn't, use following command to remove the existing subscription before recreating it:

DROP SUBSCRIPTION my_subscription;

Conclusion

In this tutorial you've successfully installed PostgreSQL 10 on two Ubuntu 18.04 servers and configured logical replication between them.

You now have the required knowledge to experiment with horizontal read scaling, high availability, and the geographical distribution of your PostgreSQL database by adding additional replica servers.

To learn more about logical replication in PostgreSQL 10, you can read the chapter on the topic on the official PostgreSQL documentation, as well as the manual entries on the CREATE PUBLICATION and CREATE SUBSCRIPTION commands.

How to Set Up SSH Keys on Debian 9

2018-08-31T17:42:17Z

Introduction

SSH, or secure shell, is an encrypted protocol used to administer and communicate with servers. When working with a Debian server, chances are you will spend most of your time in a terminal session connected to your server through SSH.

In this guide, we'll focus on setting up SSH keys for a vanilla Debian 9 installation. SSH keys provide an easy, secure way of logging into your server and are recommended for all users.

Step 1 — Create the RSA Key Pair

The first step is to create a key pair on the client machine (usually your computer):

ssh-keygen

By default ssh-keygen will create a 2048-bit RSA key pair, which is secure enough for most use cases (you may optionally pass in the -b 4096 flag to create a larger 4096-bit key).

After entering the command, you should see the following output:

OutputGenerating public/private rsa key pair.
Enter file in which to save the key (/your_home/.ssh/id_rsa):

Press enter to save the key pair into the .ssh/ subdirectory in your home directory, or specify an alternate path.

If you had previously generated an SSH key pair, you may see the following prompt:

Output
/home/your_home/.ssh/id_rsa already exists.
Overwrite (y/n)?

If you choose to overwrite the key on disk, you will not be able to authenticate using the previous key anymore. Be very careful when selecting yes, as this is a destructive process that cannot be reversed.

You should then see the following prompt:

OutputEnter passphrase (empty for no passphrase):

Here you optionally may enter a secure passphrase, which is highly recommended. A passphrase adds an additional layer of security to prevent unauthorized users from logging in. To learn more about security, consult our tutorial on How To Configure SSH Key-Based Authentication on a Linux Server.

You should then see the following output:

Output
Your identification has been saved in /your_home/.ssh/id_rsa.
Your public key has been saved in /your_home/.ssh/id_rsa.pub.
The key fingerprint is:
a9:49:2e:2a:5e:33:3e:a9:de:4e:77:11:58:b6:90:26 username@remote_host
The key's randomart image is:
+--[ RSA 2048]----+
|     ..o         |
|   E o= .        |
|    o. o         |
|        ..       |
|      ..S        |
|     o o.        |
|   =o.+.         |
|. =++..          |
|o=++.            |
+-----------------+

You now have a public and private key that you can use to authenticate. The next step is to place the public key on your server so that you can use SSH-key-based authentication to log in.

Step 2 — Copy the Public Key to Debian Server

The quickest way to copy your public key to the Debian host is to use a utility called ssh-copy-id. Due to its simplicity, this method is highly recommended if available. If you do not have ssh-copy-id available to you on your client machine, you may use one of the two alternate methods provided in this section (copying via password-based SSH, or manually copying the key).

Copying Public Key Using `ssh-copy-id`

The ssh-copy-id tool is included by default in many operating systems, so you may have it available on your local system. For this method to work, you must already have password-based SSH access to your server.

To use the utility, you simply need to specify the remote host that you would like to connect to and the user account that you have password SSH access to. This is the account to which your public SSH key will be copied.

The syntax is:

ssh-copy-id username@remote_host

You may see the following message:

Output
The authenticity of host '203.0.113.1 (203.0.113.1)' can't be established.
ECDSA key fingerprint is fd:fd:d4:f9:77:fe:73:84:e1:55:00:ad:d6:6d:22:fe.
Are you sure you want to continue connecting (yes/no)? yes

This means that your local computer does not recognize the remote host. This will happen the first time you connect to a new host. Type "yes" and press ENTER to continue.

Next, the utility will scan your local account for the id_rsa.pub key that we created earlier. When it finds the key, it will prompt you for the password of the remote user's account:

Output/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
username@203.0.113.1's password:

Type in the password (your typing will not be displayed for security purposes) and press ENTER. The utility will connect to the account on the remote host using the password you provided. It will then copy the contents of your ~/.ssh/id_rsa.pub key into a file in the remote account's home ~/.ssh directory called authorized_keys.

You should see the following output:

OutputNumber of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@203.0.113.1'"
and check to make sure that only the key(s) you wanted were added.

At this point, your id_rsa.pub key has been uploaded to the remote account. You can continue on to Step 3.

Copying Public Key Using SSH

If you do not have ssh-copy-id available, but you have password-based SSH access to an account on your server, you can upload your keys using a conventional SSH method.

We can do this by using the cat command to read the contents of the public SSH key on our local computer and piping that through an SSH connection to the remote server.

On the other side, we can make sure that the ~/.ssh directory exists and has the correct permissions under the account we’re using.

We can then output the content we piped over into a file called authorized_keys within this directory. We’ll use the >> redirect symbol to append the content instead of overwriting it. This will let us add keys without destroying previously added keys.

The full command looks like this:

cat ~/.ssh/id_rsa.pub | ssh username@remote_host "mkdir -p ~/.ssh && touch ~/.ssh/authorized_keys && chmod -R go= ~/.ssh && cat >> ~/.ssh/authorized_keys"

You may see the following message:

Output
The authenticity of host '203.0.113.1 (203.0.113.1)' can't be established.
ECDSA key fingerprint is fd:fd:d4:f9:77:fe:73:84:e1:55:00:ad:d6:6d:22:fe.
Are you sure you want to continue connecting (yes/no)? yes

This means that your local computer does not recognize the remote host. This will happen the first time you connect to a new host. Type "yes" and press ENTER to continue.

Afterwards, you should be prompted to enter the remote user account password:

Output
username@203.0.113.1's password:

After entering your password, the content of your id_rsa.pub key will be copied to the end of the authorized_keys file of the remote user's account. Continue on to Step 3 if this was successful.

Copying Public Key Manually

If you do not have password-based SSH access to your server available, you will have to complete the above process manually.

We will manually append the content of your id_rsa.pub file to the ~/.ssh/authorized_keys file on your remote machine.

To display the content of your id_rsa.pub key, type this into your local computer:

cat ~/.ssh/id_rsa.pub

You will see the key's content, which should look something like this:

Outputssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqql6MzstZYh1TmWWv11q5O3pISj2ZFl9HgH1JLknLLx44+tXfJ7mIrKNxOOwxIxvcBF8PXSYvobFYEZjGIVCEAjrUzLiIxbyCoxVyle7Q+bqgZ8SeeM8wzytsY+dVGcBxF6N4JS+zVk5eMcV385gG3Y6ON3EG112n6d+SMXY0OEBIcO6x+PnUSGHrSgpBgX7Ks1r7xqFa7heJLLt2wWwkARptX7udSq05paBhcpB0pHtA1Rfz3K2B+ZVIpSDfki9UVKzT8JUmwW6NNzSgxUfQHGwnW7kj4jp4AT0VZk3ADw497M2G/12N0PPB5CnhHf7ovgy6nL1ikrygTKRFmNZISvAcywB9GVqNAVE+ZHDSCuURNsAInVzgYo9xgJDW8wUw2o8U77+xiFxgI5QSZX3Iq7YLMgeksaO4rBJEa54k8m5wEiEE1nUhLuJ0X/vh2xPff6SQ1BL/zkOhvJCACK6Vb15mDOeCSq54Cr7kvS46itMosi/uS66+PujOO+xt/2FWYepz6ZlN70bRly57Q06J+ZJoc9FfBCbCyYH7U/ASsmY095ywPsBo1XQ9PqhnN1/YOorJ068foQDNVpm146mUpILVxmq41Cj55YKHEazXGsdBIbXWhcrRf4G2fJLRcGUr9q8/lERo9oxRm5JFX6TCmj6kmiFqv+Ow9gI0x8GvaQ== demo@test

Access your remote host using whichever method you have available.

Once you have access to your account on the remote server, you should make sure the ~/.ssh directory exists. This command will create the directory if necessary, or do nothing if it already exists:

mkdir -p ~/.ssh

Now, you can create or modify the authorized_keys file within this directory. You can add the contents of your id_rsa.pub file to the end of the authorized_keys file, creating it if necessary, using this command:

echo public_key_string >> ~/.ssh/authorized_keys

In the above command, substitute the public_key_string with the output from the cat ~/.ssh/id_rsa.pub command that you executed on your local system. It should start with ssh-rsa AAAA....

Finally, we’ll ensure that the ~/.ssh directory and authorized_keys file have the appropriate permissions set:

chmod -R go= ~/.ssh

This recursively removes all “group” and “other” permissions for the ~/.ssh/ directory.

If you’re using the root account to set up keys for a user account, it’s also important that the ~/.ssh directory belongs to the user and not to root:

chown -R sammy:sammy ~/.ssh

In this tutorial our user is named sammy but you should substitute the appropriate username into the above command.

We can now attempt passwordless authentication with our Debian server.

Step 3 — Authenticate to Debian Server Using SSH Keys

If you have successfully completed one of the procedures above, you should be able to log into the remote host without the remote account's password.

The basic process is the same:

ssh username@remote_host

If this is your first time connecting to this host (if you used the last method above), you may see something like this:

Output
The authenticity of host '203.0.113.1 (203.0.113.1)' can't be established.
ECDSA key fingerprint is fd:fd:d4:f9:77:fe:73:84:e1:55:00:ad:d6:6d:22:fe.
Are you sure you want to continue connecting (yes/no)? yes

This means that your local computer does not recognize the remote host. Type "yes" and then press ENTER to continue.

If you did not supply a passphrase for your private key, you will be logged in immediately. If you supplied a passphrase for the private key when you created the key, you will be prompted to enter it now (note that your keystrokes will not display in the terminal session for security). After authenticating, a new shell session should open for you with the configured account on the Debian server.

If key-based authentication was successful, continue on to learn how to further secure your system by disabling password authentication.

Step 4 — Disable Password Authentication on your Server

If you were able to log into your account using SSH without a password, you have successfully configured SSH-key-based authentication to your account. However, your password-based authentication mechanism is still active, meaning that your server is still exposed to brute-force attacks.

Before completing the steps in this section, make sure that you either have SSH-key-based authentication configured for the root account on this server, or preferably, that you have SSH-key-based authentication configured for a non-root account on this server with sudo privileges. This step will lock down password-based logins, so ensuring that you will still be able to get administrative access is crucial.

Once you've confirmed that your remote account has administrative privileges, log into your remote server with SSH keys, either as root or with an account with sudo privileges. Then, open up the SSH daemon's configuration file:

sudo nano /etc/ssh/sshd_config

Inside the file, search for a directive called PasswordAuthentication. This may be commented out. Uncomment the line and set the value to "no". This will disable your ability to log in via SSH using account passwords:

/etc/ssh/sshd_config

...
PasswordAuthentication no
...

Save and close the file when you are finished by pressing CTRL + X, then Y to confirm saving the file, and finally ENTER to exit nano. To actually implement these changes, we need to restart the sshd service:

sudo systemctl restart ssh

As a precaution, open up a new terminal window and test that the SSH service is functioning correctly before closing this session:

ssh username@remote_host

Once you have verified your SSH service, you can safely close all current server sessions.

The SSH daemon on your Debian server now only responds to SSH keys. Password-based authentication has successfully been disabled.

Conclusion

You should now have SSH-key-based authentication configured on your server, allowing you to sign in without providing an account password.

If you'd like to learn more about working with SSH, take a look at our SSH Essentials Guide.

A Rede do Kubernetes nos Bastidores

2018-08-27T20:43:42Z

Introdução

O Kubernetes é um poderoso sistema de orquestração de container que pode gerenciar o deployment e a operação de aplicações containerizadas em um cluster de servidores. Além de coordenar as cargas de trabalho do container, o Kubernetes fornece a infraestrutura e as ferramentas necessárias para manter a conectividade de rede entre suas aplicações e serviços.

A Documentação de Rede do Cluster do Kubernetes afirma que os requisitos básicos de uma rede Kubernetes são:

todos os containers podem se comunicar com todos os outros containers sem NAT
todos os nodes podem se comunicar com todos os containers (e vice-versa) sem NAT
o IP com o qual um container se vê é o mesmo IP que os outros o veem

Neste artigo, discutiremos como o Kubernetes satisfaz esses requisitos de rede dentro de um cluster: como os dados se movem dentro de um pod, entre pods e entre nodes.

Também mostraremos como um Serviço do Kubernetes pode fornecer um único endereço IP estático e uma entrada de DNS para uma aplicação, facilitando a comunicação com serviços que podem ser distribuídos entre vários pods de dimensionamento e deslocamento constantes.

Se você não estiver familiarizado com a terminologia dos pods e nodes do Kubernetes ou com outros itens básicos, nosso artigo An Introduction to Kubernetes cobre a arquitetura geral e os componentes envolvidos.

Primeiro, vamos dar uma olhada na situação da rede dentro de um único pod.

A Rede do Pod

No Kubernetes, um pod é a unidade mais básica de organização: um grupo de containers fortemente acoplados que estão todos intimamente relacionados e executam uma única função ou serviço.

Em termos de rede, o Kubernetes trata pods de maneira semelhante a uma máquina virtual tradicional ou a um único host físico: cada pod recebe um único endereço IP exclusivo, e todos os containers dentro do pod compartilham esse endereço e se comunicam entre si através da interface de loopback lo usando o nome de host localhost. Isso é conseguido atribuindo todos os containers do pod à mesma pilha de rede.

Essa situação deve parecer familiar para qualquer pessoa que fez o deploy de vários serviços em um único host antes dos dias da containerização. Todos os serviços precisam usar uma porta exclusiva para ouvir, mas, por outro lado, a comunicação é descomplicada e tem pouca sobrecarga.

A Rede de Pod para Pod

A maioria dos clusters do Kubernetes precisará fazer deploy de vários pods por node. A comunicação de pod para pod pode ocorrer entre dois pods no mesmo node ou entre dois nodes diferentes.

Comunicação Pod a Pod em um Node

Em um único node, você pode ter vários pods que precisam se comunicar diretamente uns com os outros. Antes de rastrearmos a rota de um pacote entre os pods, vamos analisar a configuração de rede de um node. O diagrama a seguir fornece uma visão geral, que abordaremos em detalhes:

Cada node tem uma interface de rede – eth0 neste exemplo – anexada à rede de clusters do Kubernetes. Essa interface fica dentro do namespace de rede root do node. Este é o namespace padrão para dispositivos de rede no Linux.

Assim como os namespaces de processo permitem que os containers isolem as aplicações em execução umas das outras, namespaces de rede isolam dispositivos de rede tais como interfaces e bridges. Cada pod em um node é atribuído ao seu próprio namespace de rede isolado.

Os namespaces de pod são conectados de volta ao namespace root com um par ethernet virtual, essencialmente um pipe entre os dois namespaces com uma interface em cada extremidade (aqui estamos utilizando veth1 no namespace root e eth0 dentro do pod).

Finalmente, os pods são conectados entre si e à interface eth0 do node através de uma bridge, br0 (seu node pode usar algo como cbr0 ou docker0). Uma bridge funciona essencialmente como um switch Ethernet físico, usando ARP (protocolo de resolução de endereço) ou roteamento baseado em IP para procurar outras interfaces locais para onde direcionar o tráfego.

Agora vamos rastrear um pacote do pod1 para o pod2:

pod1 cria um pacote com o IP do pod2 como seu destino
O pacote trafega pelo par de ethernet virtual para o namespace root da rede
O pacote continua até a bridge br0
Como o pod de destino está no mesmo node, a bridge envia o pacote para o par de ethernet virtual do pod2
O pacote trafega através do par de ethernet virtual, no namespace de rede do pod2 e na interface de rede eth0 do pod.

Agora que rastreamos um pacote de pod para pod dentro de um node, vamos ver como o tráfego do pod viaja entre nodes.

Comunicação Pod para Pod entre dois Nodes

Como cada pod em um cluster tem um IP exclusivo e cada pod pode se comunicar diretamente com todos os outros pods, um pacote que se move entre os pods em dois nodes distintos é muito semelhante ao cenário anterior.

Vamos rastrear um pacote do pod1 para o pod3, que está em outro node:

pod1 cria um pacote com o IP do pod3 como seu destino
O pacote trafega pelo par de ethernet virtual para o namespace root da rede
O pacote continua até a bridge br0
A bridge não encontra nenhuma interface local para onde rotear, assim o pacote é enviado para a rota padrão via eth0
Opcional: se o seu cluster exigir uma sobreposição de rede para rotear corretamente os pacotes para os nodes, o pacote poderá ser encapsulado em um pacote VXLAN (ou outra técnica de virtualização de rede) antes de ir para a rede. Alternativamente, a própria rede pode ser configurada com as rotas estáticas adequadas, nesse caso, o pacote trafega para eth0 e sai da rede inalterado.
O pacote entra na rede do cluster e é roteado para o node correto.
O pacote entra no node de destino na eth0
Opcional: se o seu pacote foi encapsulado, ele será desencapsulado neste momento
O pacote continua para a bridge br0
A bridge encaminha o pacote para o par de ethernet virtual do pod de destino
O pacote passa pelo par de ethernet virtual para a interface eth0 do pod

Agora que estamos familiarizados com a forma como os pacotes são roteados por meio dos endereços IP do pod, vamos dar uma olhada nos serviços do Kubernetes e em como eles se baseiam nessa infraestrutura.

A Rede de Pod para Serviço

Seria difícil enviar tráfego para uma aplicação específica usando apenas IPs de pod, pois a natureza dinâmica de um cluster do Kubernetes significa que os pods podem ser movidos, reiniciados, atualizados ou redimensionados para dentro e para fora. Além disso, alguns serviços terão muitas réplicas, por isso precisamos de alguma forma de balancear a carga entre eles.

O Kubernetes resolve esse problema com os Serviços. Um Serviço é um objeto da API que mapeia um único IP virtual (VIP) para um conjunto de IPs de pod. Além disso, o Kubernetes fornece uma entrada de DNS para o nome de cada serviço e IP virtual, para que os serviços possam ser facilmente acessados por nome.

O mapeamento de IPs virtuais para IPs de pods dentro do cluster é coordenado pelo processo kube-proxy em cada node. Esse processo configura ou o iptables ou IPVS para traduzir automaticamente os VIPs em IPs de pods antes de enviar o pacote para a rede do cluster. Conexões individuais são rastreadas para que os pacotes possam ser devidamente decodificados quando retornarem. O IPVS e o iptables podem fazer o balanceamento de carga de um único IP virtual de serviço em vários IPs de pods, embora o IPVS tenha muito mais flexibilidade nos algoritmos de balanceamento de carga que ele pode usar.

Nota: Este processo de rastreamento de tradução e de conexão acontece inteiramente no kernel do Linux. O kube-proxy lê a API do Kubernetes e atualiza o ip no iptables e IPVS, mas ele não está no caminho dos dados para pacotes individuais. Isso é mais eficiente e de melhor desempenho do que as versões anteriores do kube-proxy, que funcionava como um proxy de mando do usuário.

Vamos seguir a rota que um pacote leva de um pod, pod1 novamente, para um serviço, service1:

pod1 cria um pacote com o IP do service1 como seu destino
O pacote trafega pelo par de ethernet virtual para o namespace root da rede
O pacote continua até a bridge br0
A bridge não encontra nenhuma interface local para onde rotear o pacote, assim o pacote é enviado para a rota padrão via eth0
Iptables ou IPVS, configurados pelo kube-proxy, acham o IP de destino do pacote e o traduzem de um IP virtual para um dos IPs do pod de serviço, usando quaisquer algoritmos de balanceamento de carga disponíveis ou especificados
Opcional: seu pacote pode ser encapsulado neste ponto, como discutido na seção anterior
O pacote entra na rede do cluster e é roteado para o node correto.
O pacote entra no node de destino na eth0
Opcional: se o seu pacote foi encapsulado, ele será desencapsulado neste momento
O pacote continua até a bridge br0
O pacote é enviado para o par de ethernet virtual via veth1
O pacote passa pelo par de ethernet virtual e entra no namespace de rede do pod através de sua interface de rede eth0

Quando o pacote retorna para o node1, a tradução de VIP para IP do pod será revertida, e o pacote retornará através da bridge e da interface virtual para o pod correto.

Conclusão

Neste artigo, analisamos a infraestrutura de rede interna de um cluster do Kubernetes. Discutimos os blocos construtivos que compõem a rede e detalhamos a jornada salto-por-salto de pacotes em diferentes cenários.

Para mais informações sobre o Kubernetes, dê uma olhada na tag para nossos tutoriais de Kubernetes e a documentação oficial do Kubernetes.

Como Inspecionar a Rede do Kubernetes

2018-08-27T20:40:28Z

Introducão

O Kubernetes é um sistema de orquestração de container que pode gerenciar aplicações containerizadas em um cluster de nodes de servidores. A manutenção da conectividade de rede entre todos os containers em um cluster requer algumas técnicas avançadas de rede. Neste artigo vamos cobrir brevemente algumas ferramentas e técnicas para inspecionar essa configuração de rede.

Estas ferramentas podem ser úteis se você estiver debugando problemas de conectividade, investigando problemas de taxa de transferência de rede, ou explorando o Kubernetes para aprender como ele funciona.

Se você quiser aprender mais sobre o Kubernetes em geral, nosso guia An Introduction to Kubernetes cobre o básico. Para uma visão específica de rede do Kubernetes, por favor leia Kubernetes Networking Under the Hood.

Começando

Este tutorial irá assumir que você tem um cluster Kubernetes, com o kubectl instalado localmente e configurado para se conectar ao cluster.

As seções seguintes contém muitos comandos que se destinam a serem executados em um node do Kubernetes. Eles se parecerão com isso:

echo 'este é um comando de node'

Comandos que devem ser executados em sua máquina local terão a seguinte aparência:

echo 'este é um comando local'

Nota: A maioria dos comandos neste tutorial precisará ser executada como usuário root. Se em vez disso você usar um usuário habilitado para o sudo em seus nodes de Kubernetes, por favor adicione sudo para executar comandos quando necessário.

Encontrando o IP do Cluster de um Pod

Para encontrar o endereço IP de um pod do Kubermetes, utilize o comando kubectl get pod em sua máquina local, com a opção -o wide. Esta oção irá listar mais informações, incluindo o node onde o pod reside, e o IP do cluster do pod.

kubectl get pod -o wide

Output
NAME                           READY     STATUS    RESTARTS   AGE       IP            NODE
hello-world-5b446dd74b-7c7pk   1/1       Running   0          22m       10.244.18.4   node-one
hello-world-5b446dd74b-pxtzt   1/1       Running   0          22m       10.244.3.4    node-two

A coluna IP irá conter o endereço IP local do cluster para cada pod.

Se você não vir o pod que está procurando, certifique-se de que você está no namespace certo. Você pode listar todos os pods em todos os namespaces adicionando o flag --all-namespaces.

Encontrando o IP de um Serviço

Você pode também encontrar o IP de um serviço utilizando o kubectl. Neste caso iremos listar todos os serviços em todos os namespaces:

kubectl get service --all-namespaces

Output
Output
NAMESPACE     NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes                 ClusterIP   10.32.0.1       <none>        443/TCP         6d
kube-system   csi-attacher-doplugin      ClusterIP   10.32.159.128   <none>        12345/TCP       6d
kube-system   csi-provisioner-doplugin   ClusterIP   10.32.61.61     <none>        12345/TCP       6d
kube-system   kube-dns                   ClusterIP   10.32.0.10      <none>        53/UDP,53/TCP   6d
kube-system   kubernetes-dashboard       ClusterIP   10.32.226.209   <none>        443/TCP         6d

O IP do serviço pode ser encontrado na coluna CLUSTER-IP.

Encontrando e Inserindo Namespaces de Rede do Pod

Cada pod do Kubernetes é atribuído ao seu próprio namespace de rede. Namespaces de rede (ou netns) são primitivas de rede do Linux que fornecem isolação entre dispositivos de rede.

Isto pode ser útil para executar comandos a partir do netns do pod, para verificar resolução de DNS ou conectividade geral de rede. Para fazer isto, precisamos primeiro olhar para o ID de processo de um dos containers em um pod. Para o Docker, podemos fazer isto com uma série de dois comandos. Primeiro, liste os containers que estão executando em um node:

docker ps

Output
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
173ee46a3926        gcr.io/google-samples/node-hello        "/bin/sh -c 'node se…"   9 days ago          Up 9 days                               k8s_hello-world_hello-world-5b446dd74b-pxtzt_default_386a9073-7e35-11e8-8a3d-bae97d2c1afd_0
11ad51cb72df        k8s.gcr.io/pause-amd64:3.1              "/pause"                 9 days ago          Up 9 days                               k8s_POD_hello-world-5b446dd74b-pxtzt_default_386a9073-7e35-11e8-8a3d-bae97d2c1afd_0
. . .

Encontre o container ID ou name de qualquer container no pod que você está interessado. Na saída acima estamos mostrando dois containers:

O primeiro container é o app hello-world executando no pod hello-world
O segundo é um container pause executando no pod hello-world. Este container existe apenas para manter o namespace de rede do pod

Para obter o ID de processo de um dos containers, tome nota do container ID ou name, e utilize-o no seguinte comando docker:

docker inspect --format '{{ .State.Pid }}' container-id-or-name

Output
14552

Um ID de processo (ou PID) será a saída. Agora podemos utilizar o programa nsenter para executar um comando no namespace de rede do processo:

nsenter -t your-container-pid -n ip addr

Certifique-se de utilizar seu próprio PID, e substitua ip addr pelo comando que você gostaria de executar dentro do namespace de rede do pod.

Nota: Uma vantagem de se utilizar nsenter para executar comandos no namespace do pod – versus a utilização de algo como docker exec – é que você tem acesso a todos os comandos disponíveis no node, em vez do conjunto de comandos gralmente limitados instalados em containers.

Encontrando a Interface Ethernet Virtual de um Pod

Cada namespace de rede do pod comunica-se com o netns raiz do node através de um pipe ethernet virtual. No lado do node, este pipe aparece como um dispositivo que geralmente começa com veth e termina em um identificador único, tal como veth77f2275 ou veth01. Dentro do pod este pipe aparece como eth0.

Pode ser útil correlacionar qual dispositivo veth está emparelhado com um pod em particular. Para fazer isto, vamos listar todos os dispositivos de rede no node, em seguida listar os dispositivos no namespace de rede do pod. Podemos correlacionar os números dos dispositivos entre as duas listas para fazer a conexão.

Primeiro, execute ip addr no namespace de rede do pod utilizando o nsenter. Consulte a seção anterior Encontrando e Inserindo Namespaces de Rede do Pod para detlahes de como fazer isto:

nsenter -t pid-do-seu-container -n ip addr

Output
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 02:42:0a:f4:03:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.3.4/24 brd 10.244.3.255 scope global eth0
       valid_lft forever preferred_lft forever

O comando mostrará uma lista das interfaces do pod. Observe o número if11 depois de eth0@ na saída do exemplo. Isso significa que essa eth0 do pod está ligada à décima primeira interface do node. Agora execute ip addr no namespace padrão do node para listar suas interfaces:

ip addr

Output
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

. . .

7: veth77f2275@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether 26:05:99:58:0d:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::2405:99ff:fe58:db9/64 scope link
       valid_lft forever preferred_lft forever
9: vethd36cef3@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether ae:05:21:a2:9a:2b brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ac05:21ff:fea2:9a2b/64 scope link
       valid_lft forever preferred_lft forever
11: veth4f7342d@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether e6:4d:7b:6f:56:4c brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::e44d:7bff:fe6f:564c/64 scope link
       valid_lft forever preferred_lft forever

A décima primeira interface é a veth4f7342d nessa saída do exemplo. Este é o pipe ethernet virtual para o pod que estamos inevstigando.

Inspeção do Rastreamento de Conexão do Conntrack

Antes da versão 1.11, o Kubernetes usava o iptables NAT e o módulo conntrack do kernel para rastrear conexões. Para listar todas as conexões sendo rastreadas atualmente, utilize o comando conntrack:

conntrack -L

Para assitir continuamente por novas conexões, utilize o flag -E:

conntrack -E

Para listar conexões controladas pelo conntrack a um endereço de destino específico, utilize o flag -d:

conntrack -L -d 10.32.0.1

Se os seus nodes estão tendo problemas para fazer conexões confiáveis aos serviços, é possível que sua tabela de rastreamento de conexões esteja cheia e que novas conexões estejam sendo descartadas. Se é esse o caso você pode ver mensagens como as seguintes em seus logs de sistema:

/var/log/syslog


Jul 12 15:32:11 worker-528 kernel: nf_conntrack: table full, dropping packet.

Há uma configuração do sysctl para o número máximo de conexões a serem rastreadas. Você pode listar o valor atual com o seguinte comando:

sysctl net.netfilter.nf_conntrack_max

Output
net.netfilter.nf_conntrack_max = 131072

Para definir um novo valor, utilize o flag -w:

sysctl -w net.netfilter.nf_conntrack_max=198000

Para tornar essa configuração permanente, adicione-a ao arquivo sysctl.conf:

/etc/sysctl.conf


. . .
net.ipv4.netfilter.ip_conntrack_max = 198000

Inspecionando as Regras do Iptables

Antes da versão 1.11, o Kubernetes usou o iptables NAT para implementar tradução de IP virtual e o balanceamento de carga para IPs de Serviço.

Para fazer um dump de todas as regras iptables em um node, utilize o comando iptables-save:

iptables-save

Como a saída pode ser longa, você pode querer redirecionar para um arquivo (iptables-save > output.txt) ou um paginador (iptables-save | less) para avaliar suas regras mais facilmente.

Para listar apenas as regras NAT do Serviço do Kubernetes, utilize o comando iptables e o flag -L para especificar o canal correto:

iptables -t nat -L KUBE-SERVICES

Output
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  anywhere             10.32.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  anywhere             10.32.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-XGLOHA7QRQ3V22RZ  tcp  --  anywhere             10.32.226.209        /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:https
. . .

Consultando o DNS do Cluster

Uma maneira de fazer o debug da resolução de DNS do cluster é fazer o deploy de um container para debug com todas as feramentas que você precisa, em seguida utilize kubectl para executar nslookup nele. Isso é descrito na documentação oficial do Kubernetes.

Outra maneira de consultar o DNS do cluster é a utilização do dig e nsenter a partir do node. Se o dig não está instalado, pode-se instalar com o apt em distribuições Linux baseadas em Debian.

apt install dnsutils

Primeiro, encontre o IP do cluster do serviço kube-dns:

kubectl get service -n kube-system kube-dns

OutputNAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP   15d

O IP do cluster está destacado acima. Em seguida, vamos utilizar nsenter para executar o dig no namespace do container. Veja a seção Encontrando e Inserindo Namespaces de Rede do Pod para mais informações sobre isso.

nsenter -t 14346 -n dig kubernetes.default.svc.cluster.local @10.32.0.10

Este comando dig procura o nome de domínio completo do Serviço de service-name.namespace.svc.cluster.local e especifica o IP do serviço DNS do cluster (@10.32.0.10).

Olhando para os Detalhes do IPVS

A partir do Kubernetes 1.11, o kube-proxy pode configurar o IPVS para lidar com a tradução de IPs de serviços virtuais para IPs de pods. Você pode listar a tabela de tradução com ipvsadm:

ipvsadm -Ln

Output
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  100.64.0.1:443 rr
  -> 178.128.226.86:443           Masq    1      0          0
TCP  100.64.0.10:53 rr
  -> 100.96.1.3:53                Masq    1      0          0
  -> 100.96.1.4:53                Masq    1      0          0
UDP  100.64.0.10:53 rr
  -> 100.96.1.3:53                Masq    1      0          0
  -> 100.96.1.4:53                Masq    1      0          0

Para mostrar um único IP de serviço, utilize a opção -t e especifique o IP desejado:

ipvsadm -Ln -t 100.64.0.10:53

Output
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  100.64.0.10:53 rr
  -> 100.96.1.3:53                Masq    1      0          0
  -> 100.96.1.4:53                Masq    1      0          0

Conclusão

Neste artigo, analisamos alguns comandos e técnicas para explorar e inspecionar os detalhes da rede do cluster do Kubernetes. Para mais informações sobre Kubernetes, dê uma olhada na nossa tag de tutoriais de Kubernetes e na documentação oficial do Kubernetes.

How to Use Node.js and Github Webhooks to Keep Remote Projects in Sync

2018-08-17T19:07:13Z

Introduction

When working on a project with multiple developers, it can be frustrating when one person pushes to a repository and then another begins making changes on an outdated version of the code. Mistakes like these cost time, which makes it worthwhile to set up a script to keep your repositories in sync. You can also apply this method in a production environment to push hotfixes and other changes quickly.

While other solutions exist to complete this specific task, writing your own script is a flexible option that leaves room for customization in the future.

GitHub lets you configure webhooks for your repositories, which are events that send HTTP requests when events happen. For example, you can use a webhook to notify you when someone creates a pull request or pushes new code.

In this guide you will develop a Node.js server that listens for a GitHub webhook notification whenever you or someone else pushes code to GitHub. This script will automatically update a repository on a remote server with the most recent version of the code, eliminating the need to log in to a server to pull new commits.

Prerequisites

To complete this tutorial, you will need:

One Ubuntu 16.04 server set up by following the Ubuntu 16.04 initial server setup guide, including a non-root user with sudo privileges and a firewall.
Git installed on your local machine. You can follow the tutorial Contributing to Open Source: Getting Started with Git to install and set up Git on your computer.
Node.js and npm installed on the remote server using the official PPA, as explained explained in How To Install Node.js on Ubuntu 16.04. Installing the distro-stable version is sufficient as it provides us with the recommended version without any additional configuration.
A repository on Github that contains your project code. If you don't have a project in mind, feel free to fork this example which we'll use in the rest of the tutorial.

Step 1 — Setting Up a Webhook

We'll start by configuring a webhook for your repository. This step is important because without it, Github doesn't know what events to send when things happen, or where to send them. We'll create the webhook first, and then create the server that will respond to its requests.

Sign in to your GitHub account and navigate to the repository you wish to monitor. Click on the Settings tab in the top menu bar on your repository's page, then click Webhooks in the left navigation menu. Click Add Webhook in the right corner and enter your account password if prompted. You'll see a page that looks like this:

In the Payload URL field, enter http://your_server_ip:8080. This is the address and port of the Node.js server we'll write soon.
Change the Content type to application/json. The script we will write will expect JSON data and won't be able to understand other data types.
For Secret, enter a secret password for this webhook. You'll use this secret in your Node.js server to validate requests and make sure they came from GitHub.
For Which events would you like to trigger this webhook, select just the push event. We only need the push event since that is when code is updated and needs to be synced to our server.
Select the Active checkbox.
Review the fields and click Add webhook to create it.

The ping will fail at first, but rest assured your webhook is now configured. Now let's get the repository cloned to the server.

Step 2 — Cloning the Repository to the Server

Our script can update a repository, but it cannot handle setting up the repository initially, so we'll do that now. Log in to your server:

ssh sammy@your_server_ip

Ensure you're in your home directory. Then use Git to clone your repository. Be sure to replace sammy with your GitHub username and hello_hapi with the name of your Github project.

cd
git clone https://github.com/sammy/hello_hapi.git

This will create a new directory containing your project. You'll use this directory in the next step.

With your project cloned, you can create the webhook script.

Step 3 — Creating the Webhook Script

Let's create our server to listen for those webhook requests from GitHub. We'll write a Node.js script that launches a web server on port 8080. The server will listen for requests from the webhook, verify the secret we specified, and pull the latest version of the code from GitHub.

Navigate to your home directory:

cd ~

Create a new directory for your webhook script called NodeWebhooks:

mkdir ~/NodeWebhooks

Then navigate to the new directory:

cd ~/NodeWebhooks

Create a new file called webhook.js inside of the NodeWebhooks directory.

nano webhook.js

Add these two lines to the script:

webhook.js

var secret = "your_secret_here";
var repo = "/home/sammy/hello_hapi";

The first line defines a variable to hold the secret you created in Step 1 which verifies that requests come from GitHub. The second line defines a variable that holds the full path to the repository you want to update on your local disk. This should point to the repository you checked out in Step 2.

Next, add these lines which import the http and crypto libaries into the script. We'll use these to create our web server and hash the secret so we can compare it with what we receive from GitHub:

webhook.js

let http = require('http');
let crypto = require('crypto');

Next, include the child_process library so you can execute shell commands from your script:

webhook.js

const exec = require('child_process').exec;

Next, add this code to define a new web server that handles GitHub webhook requests and pulls down the new version of the code if it's an authentic request:

webhook.js

http.createServer(function (req, res) {
    req.on('data', function(chunk) {
        let sig = "sha1=" + crypto.createHmac('sha1', secret).update(chunk.toString()).digest('hex');

        if (req.headers['x-hub-signature'] == sig) {
            exec('cd ' + repo + ' && git pull');
        }
    });

    res.end();
}).listen(8080);

The http.createServer() function starts a web server on port 8080 which listens for incoming requests from Github. For security purposes, we validate that the secret included in the request matches the one we specified when creating the webhook in Step 1. The secret is passed in the x-hub-signature header as an SHA1-hashed string, so we hash our secret and compare it to what GitHub sends us.

If the request is authentic, we execute a shell command to update our local repository using git pull.

The completed script looks like this:

webhook.js

const secret = "your_secret_here";
const repo = "~/your_repo_path_here/";

const http = require('http');
const crypto = require('crypto');
const exec = require('child_process').exec;

http.createServer(function (req, res) {
    req.on('data', function(chunk) {
        let sig = "sha1=" + crypto.createHmac('sha1', secret).update(chunk.toString()).digest('hex');

        if (req.headers['x-hub-signature'] == sig) {
            exec('cd ' + repo + ' && git pull');
        }
    });

    res.end();
}).listen(8080);

If you followed the initial server setup guide, you will need to allow this web server to communicate with the outside web by allowing traffic on port 8080:

sudo ufw allow 8080/tcp

Now that our script is in place, let's make sure that it is working properly.

Step 4 - Testing the Webhook

We can test our webhook by using node to run it in the command line. Start the script and leave the process open in your terminal:

cd ~/NodeWebhooks
nodejs webhook.js

Return to your project's page on Github.com. Click on the Settings tab in the top menu bar on your repository's page, followed by clicking Webhooks in the left navigation menu. Click Edit next to the webhook you set up in Step 1. Scroll down until you see the Recent Deliveries section, as shown in the following image:

Press the three dots to the far right to reveal the Redeliver button. With the node server running, click Redeliver to send the request again. Once you confirm you want to send the request, you'll see a successful response. This is indicated by a 200 OK response code after redelivering the ping.

We can now move on to making sure our script runs in the background and starts at boot. Use CTRL+C stops the node webhook server.

Step 5 — Installing the Webhook as a Systemd Service

systemd is the task manager Ubuntu uses to control services. We will set up a service that will allow us to start our webhook script at boot and use systemd commands to manage it like we would with any other service.

Start by creating a new service file:

sudo nano /etc/systemd/system/webhook.service

Add the following configuration to the service file which tells systemd how to run the script. This tells Systemd where to find our node script and describes our service.

Make sure to replace sammy with your username.

/etc/systemd/system/webhook.service

[Unit]
Description=Github webhook
After=network.target

[Service]
Environment=NODE_PORT=8080
Type=simple
User=sammy
ExecStart=/usr/bin/nodejs /home/sammy/NodeWebhooks/webhook.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable the new service so it starts when the system boots:

sudo systemctl enable webhook.service

Now start the service:

sudo systemctl start webhook

Ensure the service is started:

sudo systemctl status webhook

You'll see the following output indicating that the service is active:

Output● webhook.service - Github webhook
   Loaded: loaded (/etc/systemd/system/webhook.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-17 19:28:41 UTC; 6s ago
 Main PID: 9912 (nodejs)
    Tasks: 6
   Memory: 7.6M
      CPU: 95ms
   CGroup: /system.slice/webhook.service
           └─9912 /usr/bin/nodejs /home/sammy/NodeWebhooks/webhook.js

You are now able to push new commits to your repository and see the changes on your server.

From your desktop machine, clone the repository:

git clone https://github.com/sammy/hello_hapi.git

Make a change to one of the files in the repository. Then commit the file and push your code to GitHub.

git add index.js
git commit -m "Update index file"
git push origin master

The webhook will fire and your changes will appear on your server.

Conclusion

You have set up a Node.js script which will automatically deploy new commits to a remote repository. You can use this process to set up additional repositories that you'd like to monitor. You could even configure it to deploy a website or application to production when you push your repository.

How to Generate a Short and Unique Digital Address for Any Location Using AngularJS and PHP

2018-08-03T20:33:22Z

Introduction

Postal addresses are usually lengthy and sometimes difficult to remember. There are a number of scenarios where a shorter address would be desirable. For example, having the ability to send a short address consisting of only a couple of characters could ensure faster delivery of emergency ambulance services. Pieter Geelen and Harold Goddijn developed the Mapcode system in 2001 to make it easy to create a short-form address for any physical address in the world.

In this tutorial, you will develop a web app that uses the Google Maps API to generate a short digital address for any address of your choice. You will do this by cloning the base code for this app from GitHub and then adding code to it that will make it fully functional. This app will also be able to retrieve the original physical address from a given mapcode.

Prerequisites

In order to complete this tutorial, you will need the following:

Access to an Ubuntu 18.04 server. This server should have a non-root user with sudo privileges and a firewall configured. To set this up, you can follow our Initial Server Setup Guide for Ubuntu 18.04.
A LAMP stack installed on your machine. This is necessary because the application that you are going to develop in this tutorial uses AngularJS and PHP, and the digital address that the application generates will be stored in a MySQL database. Follow our guide on How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu 18.04 to set this up.
Git installed on your server. You can follow the tutorial Contributing to Open Source: Getting Started with Git to install and set up Git.

Step 1 — Getting a Google API Key

In this tutorial, you will use JavaScript to create an interface to Google Maps. Google assigns API keys to enable developers to use the JavaScript API on Google Maps, which you will need to obtain and add to your web app's code.

To get your own API key, head to Google's "Get API Key" page. Click on the GET STARTED button in Step 1, and a pop-up will open as shown in the following image:

Select Maps by clicking the check box and hit CONTINUE. If you aren't already logged into a Google account, you will be asked to do so. Then, the window will ask you to provide a name for the project, which can be anything you'd like:

Following this, it will ask you to enter your billing information. Note that Google provides API keys as part of a free trial, but it requires you to set up and enable billing in order retrieve them.

After entering this information, your API key will appear on the screen. Copy and store it in a location where you can easily retrieve it, as you will need to add it to your project code later on.

After obtaining your API key, you can begin building the foundation of your application by creating a MySQL database.

Step 2 — Creating the Database

The web application described in this tutorial accepts an address from the user and generates a mapcode for it along with the latitude and longitude of the specified location. You will store this data in a MySQL database so that you can retrieve it later on just by entering the respective digital address.

Begin by opening the MySQL shell and authenticating with your password:

mysql -u root -p

At the prompt, create a database called digitaladdress using the following command:

CREATE DATABASE IF NOT EXISTS `digitaladdress`;

Next, select this new database so that you can create a table within it:

USE `digitaladdress`;

After selecting the digitaladdress database, create a table called locations within it to store the physical address, its longitude, latitude, and the mapcode that your application will create from this data. Run the following CREATE TABLE statement to create the locations table within the database:

CREATE TABLE `locations` (
  `digitaladdress` varchar(50) DEFAULT NULL,
  `state` varchar(30) DEFAULT NULL,
  `zip` varchar(30) DEFAULT NULL,
  `street` varchar(30) DEFAULT NULL,
  `town` varchar(30) DEFAULT NULL,
  `house` varchar(30) DEFAULT NULL,
  `latitude` varchar(30) DEFAULT NULL,
  `longitude` varchar(30) DEFAULT NULL,
  KEY `digitaladdress` (`digitaladdress`)
);

This table has eight columns: digitaladdress, state, zip, street, town, house, latitude, and longitude. The first column, digitaladdress, is indexed using the KEY command. Indexes in MySQL function similarly to how they work in an encyclopedia or other reference work. Any time you or your application issue a query containing a WHERE statement, MySQL reads every entry in each column, row-by-row, which can become an extremely resource-intensive process as your table accumulates more and more entries. Indexing a column like this takes the data from the column and stores it alphabetically in a separate location, which means that MySQL will not have to look through every row in the table. It only has to find the data you're looking for in the index and then jump to the corresponding row in the table.

After adding this table, exit the MySQL prompt:

exit

With your database and table set up and your Google Maps API key in hand, you're ready to create the project itself.

Step 3 — Creating the Project

As mentioned in the introduction, we will clone the base code for this project from GitHub and then add some extra code to make the application functional. The reason for this, rather than walking you through the process of creating each file and adding all the code yourself, is to speed up the process of getting the app running. It will also allow us to focus on adding and understanding the code that allows the app to communicate with both the Google Maps and Mapcode APIs.

You can find the skeleton code for the full project on this GitHub project page. Use the following git command to clone the project to your server:

git clone https://github.com/do-community/digiaddress.git

This will create a new folder called digiaddress in your home directory. Move this directory to your server's web root. If you followed the LAMP stack tutorial linked in the prerequisites, this will be the /var/www/html directory:

sudo mv digiaddress/ /var/www/html/

This project contains several PHP and JS files to which you'll add some code later on in this tutorial. To view the directory structure, first install the tree package using apt:

sudo apt install tree

Then run the tree command with the digiaddress directory given as an argument:

tree /var/www/html/digiaddress/

Outputdigiaddress/
├── README.md
├── db.php
├── fetchaddress.php
├── findaddress.php
├── generateDigitalAddress.php
├── geoimplement.php
├── index.php
└── js
    ├── createDigitialAddressApp.js
    └── findAddressApp.js

You can see from this output that the project consists of six PHP files and two JavaScript files. Together, these files create the application's two main functionalities: creating a mapcode from a physical address, and decoding a mapcode to retrieve the original physical address. The following files enable the first functionality:

index.php
geoimplement.php
generateDigitialAddress.php
db.php
createDigitialAddressApp.js

The index.php file contains the code for the application's user interface (UI), which consists of a form where users can enter a physical address. The index.php file calls the geoimplement.php file any time a user submits the form. geoimplement.php makes a call to the Google Maps API and passes the address along to it. The Google server then responds with a JSON containing the specified address's information, including its latitude and longitude. This information is then passed to the generateDigitalAddress.php file which calls the Mapcode API to obtain a mapcode for the given location, as specified by its latitude and longitude. The resulting mapcode, along with the latitude, longitude, and the physical address, are then stored in the database that you created in Step 2. db.php acts as a helper for this operation. The createDigitalAddressApp.js file performs a number of operations that control the UX elements seen in the app, including setting a marker and boundary rectangle on the Google Maps interface.

The remaining three files enable the second function of the application — that is, retrieving a physical address from a given mapcode:

findaddress.php
fetchaddress.php
findAddressApp.js

The findaddress.php file defines the application UI, which is distinct from the one defined in index.php. The application accepts a previously-generated mapcode as an input and displays the corresponding physical address stored in the database. Whenever a user submits this form, findaddress.php sends a call to fetchaddress.php which then retrieves the respective mapcode from the database. The findAddressApp.js file contains the helper code for setting a marker and a boundary rectangle on the Google Maps interface.

Test the installation by visiting http://your_server_ip/digiaddress in your browser, making sure to change your_server_ip to reflect your server's IP address.

Note: If you don't know your server's IP address, you can run the following curl command. This command will print the page content of icanhazip.com, a website that shows the IP address of the machine accessing it:

curl http://icanhazip.com

Once there, you will see this heading at the top of your browser window:

Generate Digital Address

This confirms that you have correctly downloaded the project files. With that, let us proceed with the development of the app's primary function: generating a mapcode.

Step 4 — Developing the Application's UI

While the boilerplate code for the application interface is included in the files you downloaded in the previous step, you still need to make a few changes and additions to some of these files to make the application functional and engaging for users. We will get started with updating the code to develop the application's UI.

Open the index.php file using your preferred editor. Here, we'll use nano:

nano /var/www/html/digiaddress/index.php

Look for the following line of code:

/var/www/html/digiaddress/index.php

. . .
<script async defer src="https://maps.googleapis.com/maps/api/js?key=<YOUR KEY>"></script>
. . .

Replace <YOUR KEY> with the Google API key you obtained in Step 1. After adding your API key, the line should look similar to this:

/var/www/html/digiaddress/index.php

. . .
<script async defer src="https://maps.googleapis.com/maps/api/js?key=ExampleAPIKeyH2vITfv1eIHbfka9ym634Esw7u"></script>
. . .

Next, find the following comment in the index.php file:

/var/www/html/digiaddress/index.php

. . .
            <!-- add form code here -->
. . .

We'll add a few dozen lines of code below this comment which will create a form where users can enter the address of a physical location which the application will use to generate a mapcode. Under this comment, add the following highlighted code which creates a title called Enter Address at the top of the form:

/var/www/html/digiaddress/index.php

. . .
            <!-- add form code here -->

            <div class="form-border spacing-top">
                <div class="card-header" style="background:#cc0001; color:#ffff">
                    <h5>Enter Address</h5>
                </div>
                <div class="extra-padding">
. . .

Below this, add the following HTML code. This creates a form with five text fields (along with their appropriate labels) where users will input their information:

/var/www/html/digiaddress/index.php

                . . .
                <form>
                        <div class="form-group input-group-sm">
                            <label for="state">State</label>
                            <input type="text" class="form-control rounded-0 textbox-border" id="state"
                                   placeholder="" ng-model="address.state"/>
                        </div>
                        <div class="form-group input-group-sm">
                            <label for="zip" class="animated-label">Zip</label>
                            <input type="text" class="form-control rounded-0 textbox-depth textbox-border"
                                   id="zip" ng-model="address.zip" disabled="disabled"/>
                        </div>
                        <div class="form-group input-group-sm">
                            <label for="town">Town</label>
                            <input type="text" class="form-control rounded-0 textbox-border"
                                   id="town" ng-model="address.town" disabled="disabled"/>
                        </div>
                        <div class="form-group input-group-sm">
                            <label for="street">Street</label>
                            <input type="text" class="form-control rounded-0 textbox-border" id="street"
                                   placeholder="" ng-model="address.street" disabled="disabled"/>
                        </div>
                        <div class="form-group input-group-sm">
                            <label for="house">House</label>
                            <input type="text" class="form-control rounded-0 textbox-border" id="house"
                                   placeholder="" ng-model="address.house" disabled="disabled"/>
                        </div>
                 . . .

Below the form code, add the following lines. These create two hidden controls which pass along the latitude and longitude information derived from any address submitted through the form:

/var/www/html/digiaddress/index.php

                            . . .
                            <div class="form-group input-group-sm">
                                <input type="hidden" ng-model="address.lat"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <input type="hidden" ng-model="address.long"/>
                            </div>
                            . . .

Lastly, close out this section by adding the following code. This creates a Generate button which will allow users to submit the form:

/var/www/html/digiaddress/index.php

                            . . .
                            <button type="submit" disabled="disabled" class="btn btn-color btn-block rounded-0" id="generate"
                                    style="color:#ffff;background-color: #cc0001;">Generate
                            </button>
                    </form>
                </div>
            </div>
        . . .

After adding these elements, this section of the file should match this:

/var/www/html/digiaddress/index.php

. . .
            <!-- add form code here -->

            <div class="form-border spacing-top">
                <div class="card-header" style="background:#cc0001; color:#ffff">
                    <h5>Enter Address</h5>
                </div>
                <div class="extra-padding">
                    <form>    
                            <div class="form-group input-group-sm">
                                <label for="state">State</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="state"
                                       placeholder="" ng-model="address.state"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="zip" class="animated-label">Zip</label>
                                <input type="text" class="form-control rounded-0 textbox-depth textbox-border"
                                       id="zip" ng-model="address.zip" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="town">Town</label>
                                <input type="text" class="form-control rounded-0 textbox-border "
                                       id="town" ng-model="address.town" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="street">Street</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="street"
                                       placeholder="" ng-model="address.street" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="house">House</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="house"
                                       placeholder="" ng-model="address.house" disabled="disabled"/>
                            </div>

                            <div class="form-group input-group-sm">
                                <input type="hidden" ng-model="address.lat"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <input type="hidden" ng-model="address.long"/>
                            </div>
                            <button type="submit" disabled="disabled" class="btn btn-color btn-block rounded-0" id="generate"
                                    style="color:#ffff;background-color: #cc0001;">Generate
                            </button>
                    </form>
                </div>
            </div>
            <br>
        </div>

        <!-- add google map control -->
                    . . .

Save the file by pressing CTRL+O then ENTER, and then visit the application in your browser again:

http://your_server_ip/digiaddress

You will see the newly-added form fields and Generate button, and the application should look like this:

At this point, if you enter address information into the form and try clicking the Generate button, nothing will happen. We will add the mapcode generation functionality later on, but let's first focus on making this page more visually engaging by adding a map which users can interact with.

Step 5 — Adding Google Maps Controls

When maps are displayed on a website through the Google Maps JavaScript API, they contain user interface features that allow visitors to interact with the map they see. These features are known as controls. We will continue editing the index.php file to add Google Maps controls to this app and, when finished, users will be able to view a map next to the input form, drag it around to view different locations, zoom in and out, and switch between Google's map, satellite, and street views.

Find the following comment within the index.php file:

/var/www/html/digiaddress/index.php

. . .
<!-- add google map control -->
. . .

Add the following highlighted code below this comment:

/var/www/html/digiaddress/index.php

. . .
        <!-- add google map control -->

        <div class="col-sm-8 map-align" ng-init="initMap()">
            <div id="map" class="extra-padding" style="height: 100%;
            margin-bottom: 15px;"></div>
            <label id="geocoordinates" ng-show="latlng" ng-model="lt"></label><br/>
            <label id="geoaddress" ng-show="address" ng-model="padd"></label>
            </div>
        </div>
. . .

Save the file, then visit the application in your browser again. You will see the following:

As you can see, we've successfully added a map to the application. You can drag the map around to focus on different locations, zoom in and out, and switch between the map, satellite, and street views. Looking back at the code you just added, notice that we've also added two label controls that will display the geocoordinates and the physical address that were entered on the form:

/var/www/html/digiaddress/index.php

            . . .
            <label id="geocoordinates" ng-show="latlng" ng-model="lt"></label><br/>
            <label id="geoaddress" ng-show="address" ng-model="padd"></label>
            . . .

Visit the application again in your browser and enter the name of a state in the first field. When you move your text cursor to the next field, the latitude and longitude labels don't appear, nor does the location shown on the map change to reflect the information you've entered. Let's enable these behaviors.

Step 6 — Adding Event Listeners

Adding interactive elements to an application can help to keep its users engaged. We will implement a few interactive behaviors in this application through the use of event listeners.

An event is any action that takes place on a web page. Events can be something done by a user or by the browser itself. Examples of common events are:

Clicking an HTML button
Changing the content of an input field
Changing the focus from one page element to another

An event listener is a directive that tells a program to take a certain action when a specific event takes place. In AngularJS, event listeners are defined with directives that generally follow this format:

ng-event_type=expression

In this step, we will add an event listener that helps to process the information entered by users into a mapcode whenever they submit the form. We will also add a couple more event listeners that will make the application more interactive. Specifically, we'll use these listeners to change the location shown in the application map, place a marker, and draw a rectangle around the location as users enter information into the form. We'll add these event listeners to index.php, so open that file up again if you've closed it:

nano /var/www/html/digiaddress/index.php

Scroll down to the first batch of code we added, and find the block that begins with <form>. It will look like this:

/var/www/html/digiaddress/index.php

                . . .
                    <form>
                            <div class="form-group input-group-sm">
                                <label for="state">State</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="state"
                                       placeholder="" ng-model="address.state"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="zip" class="animated-label">Zip</label>
                                <input type="text" class="form-control rounded-0 textbox-depth textbox-border"
                                       id="zip" ng-model="address.zip" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="town">Town</label>
                                <input type="text" class="form-control rounded-0 textbox-border"
                                       id="town" ng-model="address.town" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="street">Street</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="street"
                                       placeholder="" ng-model="address.street" disabled="disabled"/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="house">House</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="house"
                                       placeholder="" ng-model="address.house" disabled="disabled"/>
                            </div>
                    </form>
. . .

To begin, add the following highlighted event listener to the opening <form> tag. This code tells the app to call the processForm function whenever a user submits information through the form. processForm is defined in the createDigitalAddressApp.js file, and serves as a helper function that sends the information submitted by users to the appropriate files which then process it into a mapcode. We will take a closer look at this function in Step 7:

/var/www/html/digiaddress/index.php

                . . .
                    <form ng-submit="processForm()" class="custom-form">
                            <div class="form-group input-group-sm">
                                <label for="state">State</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="state"
                                       placeholder="" ng-model="address.state"
                            </div>
                . . .

Next, continue editing this block by adding a couple blur event listeners. A blur event occurs when a given page element loses focus. Add the following highlighted lines to the form block's input tags. These lines tell the application to call the geocodeAddress function when a user's focus shifts away from the respective form fields we created in Step 4. Note that you must also delete the slashes and greater-than signs (/>) that close out each input tag. Failing to do so will prevent the app from registering the blur events correctly:

/var/www/html/digiaddress/index.php

                . . .
                <form ng-submit="processForm()" class="custom-form">
                            <div class="form-group input-group-sm">
                                <label for="state">State</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="state"
                                       placeholder="" ng-model="address.state"
                                       ng-blur="geocodeAddress(address,'state')" required=""/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="zip" class="animated-label">Zip</label>
                                <input type="text" class="form-control rounded-0 textbox-depth textbox-border"
                                       id="zip" ng-model="address.zip" disabled="disabled"
                                       ng-blur="geocodeAddress(address,'zip')" required=""/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="town">Town</label>
                                <input type="text" class="form-control rounded-0 textbox-border"
                                       id="town" ng-model="address.town" disabled="disabled"
                                       ng-blur="geocodeAddress(address,'town')" required=""/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="street">Street</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="street"
                                       placeholder="" ng-model="address.street" disabled="disabled"
                                       ng-blur="geocodeAddress(address,'street')" required=""/>
                            </div>
                            <div class="form-group input-group-sm">
                                <label for="house">House</label>
                                <input type="text" class="form-control rounded-0 textbox-border" id="house"
                                       placeholder="" ng-model="address.house" disabled="disabled"
                                       ng-blur="geocodeAddress(address,'house')" required=""/>
                            </div>
. . .

The first of these new lines — ng-blur="geocodeAddress(address,'state')" required=""/> — translates to "When the user's focus shifts away from the 'state' field, call the geocodeAddress function." The other new lines also call geocodeAddress, albeit when the user's focus shifts away from their respective fields.

As with the processForm function, geocodeAddress is declared in the createDigitalAddressApp.js file, but there isn't yet any code in that file that defines it. We will complete this function so that it places a marker and draws a rectangle on the application map after these blur events occur to reflect the information entered into the form. We'll also add some code that takes the address information and processes it into a mapcode.

Save and close the index.php file (press CTRL+X, Y, then ENTER) and then open thecreateDigitalAddressApp.js file:

nano /var/www/html/digiaddress/js/createDigitalAddressApp.js

In this file, find the following line:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
$scope.geocodeAddress = function (address, field) {
. . .

This line is where we declare the geocodeAddress function. A few lines below this, we declare a variable named fullAddress which constructs a human-readable mailing address from the information entered by a user into the application's form fields. This is done through a series of if statements:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
var fullAddress = "";

    if (address ['house']) {
        angular.element(document.getElementById('generate'))[0].disabled = false;
            fullAddress = address ['house'] + ",";
                }
    if (address ['town']) {
        angular.element(document.getElementById('street'))[0].disabled = false;
            fullAddress = fullAddress + address ['town'] + ",";
    }
    if (address ['street']) {
        angular.element(document.getElementById('house'))[0].disabled = false;
            fullAddress = fullAddress + address ['street'] + ",";
    }
    if (address ['state']) {
        angular.element(document.getElementById('zip'))[0].disabled = false;
            fullAddress = fullAddress + address ['state'] + " ";
    }
    if (address ['zip']) {
        angular.element(document.getElementById('town'))[0].disabled = false;
            fullAddress = fullAddress + address ['zip'];
    }
. . .

Directly after these lines is the following comment:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
// add code for locating the address on Google maps
. . .

Underneath this comment, add the following line which checks whether fullAddress is any value other than null:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                . . .
                if (fullAddress !== "") {
                . . .

Add the following code below this line. This code submits the information entered into the form to the geoimplement.php file using the HTTP POST method if fullAddress is not null:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                    . . .
                    $http({
                        method: 'POST',
                        url: 'geoimplement.php',
                        data: {address: fullAddress},
                        headers: {'Content-Type': 'application/x-www-form-urlencoded'}

                    }).then(function successCallback(results) {
                    . . .

Next, add the following line which checks whether the PHP call was returned successfully:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                        . . .
                        if (results.data !== "false") {
                        . . .

If the PHP call was successfully returned, we'll be able to process the result. Add the following line, which removes any boundary rectangle that may have been previously drawn on the map by calling the removeRectangle function, which is defined at the top of the createDigitalAddressApp.js file:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                            . . .
                            removeRectangle();
                            . . .

Under the removeRectangle(); line, add the following four lines which will create a marker pointing to the new location on the map control:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                            . . .
                            new google.maps.Marker({
                                map: locationMap,
                                position: results.data.geometry.location
                            });
                            . . .

Then add the following code, which obtains the latitude and longitude information from the result and displays it with the two HTML labels we created in the index.php file in Step 5:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                            . . .
                            lat = results.data.geometry.location.lat;
                            lng = results.data.geometry.location.lng;

                            $scope.address.lat = lat;
                            $scope.address.lng = lng;

                            geoCoordLabel = angular.element(document.querySelector('#geocoordinates'));
                            geoCoordLabel.html("Geo Coordinate: " + lat + "," + lng);

                            geoAddressLabel = angular.element(document.querySelector('#geoaddress'));
                            geoAddressLabel.html("Geo Address: " + fullAddress);

                            $scope.latlng = true;
                            . . .

Lastly, below these lines, add the following content. This code creates a viewport which marks a new boundary rectangle on the map:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                            . . .
                            if (results.data.geometry.viewport) {

                                rectangle = new google.maps.Rectangle({
                                    strokeColor: '#FF0000',
                                    strokeOpacity: 0.8,
                                    strokeWeight: 0.5,
                                    fillColor: '#FF0000',
                                    fillOpacity: 0.35,
                                    map: locationMap,
                                    bounds: {
                                        north: results.data.geometry.viewport.northeast.lat,
                                        south: results.data.geometry.viewport.southwest.lat,
                                        east: results.data.geometry.viewport.northeast.lng,
                                        west: results.data.geometry.viewport.southwest.lng
                                    }
                                });

                                var googleBounds = new google.maps.LatLngBounds(results.data.geometry.viewport.southwest, results.data.geometry.viewport.northeast);

                                locationMap.setCenter(new google.maps.LatLng(lat, lng));
                                locationMap.fitBounds(googleBounds);
                            }
                        } else {
                            errorLabel = angular.element(document.querySelector('#lt'));
                            errorLabel.html("Place not found.");
                            $scope.latlng = true;
                            removeRectangle();
                        }

                    }, function errorCallback(results) {
                       console.log(results);
                    });
                }
                . . .

After adding this content, this section of the file will look like this:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

                . . .
                // add code for locating the address on Google maps
                if (fullAddress !== "") {
                    $http({
                        method: 'POST',
                        url: 'geoimplement.php',
                        data: {address: fullAddress},
                        headers: {'Content-Type': 'application/x-www-form-urlencoded'}

                    }).then(function successCallback(results) {

                        if (results.data !== "false") {
                            removeRectangle();

                            new google.maps.Marker({
                                map: locationMap,
                                position: results.data.geometry.location
                            });

                            lat = results.data.geometry.location.lat;
                            lng = results.data.geometry.location.lng;

                            $scope.address.lat = lat;
                            $scope.address.lng = lng;

                            geoCoordLabel = angular.element(document.querySelector('#geocoordinates'));
                            geoCoordLabel.html("Geo Coordinate: " + lat + "," + lng);

                            geoAddressLabel = angular.element(document.querySelector('#geoaddress'));
                            geoAddressLabel.html("Geo Address: " + fullAddress);

                            $scope.latlng = true;

                            if (results.data.geometry.viewport) {

                                rectangle = new google.maps.Rectangle({
                                    strokeColor: '#FF0000',
                                    strokeOpacity: 0.8,
                                    strokeWeight: 0.5,
                                    fillColor: '#FF0000',
                                    fillOpacity: 0.35,
                                    map: locationMap,
                                    bounds: {
                                        north: results.data.geometry.viewport.northeast.lat,
                                        south: results.data.geometry.viewport.southwest.lat,
                                        east: results.data.geometry.viewport.northeast.lng,
                                        west: results.data.geometry.viewport.southwest.lng
                                    }
                                });

                                var googleBounds = new google.maps.LatLngBounds(results.data.geometry.viewport.southwest, results.data.geometry.viewport.northeast);

                                locationMap.setCenter(new google.maps.LatLng(lat, lng));
                                locationMap.fitBounds(googleBounds);
                            }
                        } else {
                            errorLabel = angular.element(document.querySelector('#lt'));
                            errorLabel.html("Place not found.");
                            $scope.latlng = true;
                            removeRectangle();
                        }

                    }, function errorCallback(results) {
                       console.log(results);
                    });
                }
                . . .

Save the file, but keep it open for now. If you were to visit the application in your browser again, you wouldn't see any new changes to its appearance or behavior. Likewise, if you were to enter an address and click on the Generate button, the application still would not generate or display a mapcode. This is because we must still edit a few files before the mapcode functionality will work. Let's continue to make these changes, and also take a closer look at how these mapcodes are generated.

Step 7 — Understanding Mapcode Generation

While still looking at the createDigitalAddressApp.js file, scroll past the section of code that you added in the previous step to find the code that takes the information submitted through the form and process it into a unique mapcode. Whenever a user clicks the Generate button, the code within the index.php file submits the form and calls the processForm function, which is defined here in createDigitalAddressApp.js:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
$scope.processForm = function () {
. . .

processForm then makes an HTTP POST to the generateDigitalAddress.php file:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
$http({
    method: 'POST',
    url: 'generateDigitalAddress.php',
    data: $scope.address,
    headers: {'Content-Type': 'application/x-www-form-urlencoded'}
}).then(function (response) {
. . .

The Stichting Mapcode Foundation provides the API that generates mapcodes from physical addresses as a free web service. To understand how this call to the Mapcode web service works, close createDigitalAddressApp.js and open the generateDigitialAddress.php file:

nano /var/www/html/digiaddress/generateDigitalAddress.php

At the top of the file, you'll see the following:

/var/www/html/digiaddress/generateDigitalAddress.php

<?php
include("db.php");
. . .

The line reading include("db.php"); tells PHP to include all the text, code, and markup from the db.php file within the generateDigitalAddress.php file. db.php holds the login credentials for the MySQL database you created in Step 2, and by including it within generateDigitalAddress.php, we can add any address information submitted through the form to the database.

Below this include statement are a few more lines that obtain the latitude and longitude information based on the request submitted by createDigitalAddressApp.js:

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
$data = json_decode(file_get_contents("php://input"));
$lat = $data->lat;
$long = $data->lng;
. . .

Look for the following comment in generateDigitalAddress.php file.

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
// call to mapcode web service
. . .

Add the following line of code below this comment. This code makes a call the Mapcode API, sending lat and long as parameters.

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
// call to mapcode web service
$digitaldata = file_get_contents("https://api.mapcode.com/mapcode/codes/".$lat.",".$long."?include=territory,alphabet&allowLog=true&client=web");
. . .

The web service returns the JSON data which was assigned to digitaldata, and the following statement decodes that JSON:

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
$digitalAddress["status"] = json_decode($digitaldata, TRUE)['local']['territory']." ".json_decode($digitaldata, TRUE)['local']['mapcode'];
. . .

This returns a mapcode for the user-specified location. The following lines then store this information in the database:

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
$obj = new databaseConnection();

$conn = $obj->dbConnect();

$obj->insertLocation($conn, $digitalAddress["status"],$data->state,$data->zip,$data->street,$data->town,$data->house,$lat,$long);
. . .

Then, the final line echoes the mapcode back to the caller function:

/var/www/html/digiaddress/generateDigitalAddress.php

. . .
echo json_encode($digitalAddress);

Save and close this file, then reopen createDigitalAddressApp.js again:

nano /var/www/html/digiaddress/js/createDigitalAddressApp.js

When a mapcode has been retrieved successfully, the following lines in the createDigitalAddressApp.js file displays it to the user in a dialog box:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

. . .
digiAddress = response.data.status;
. . .
$('#digitalAddressDialog').modal('show');
. . .

Although you did add a new line of code to generateDigitalAddress.php, you still won't see any functional changes when you visit and interact with the app in your browser. This is because you've not yet added your Google API key to the geoimplement.php file, which makes the actual call to the Google Maps API.

Step 8 — Enabling Calls to the Google Maps API

This application depends on the Google Maps API to translate a physical address into the appropriate latitude and longitude coordinates. These are then passed on to the Mapcode API which uses them to generate a mapcode. Consequently, if the application is unable to communicate with the Google Maps API to generate the location's latitude and longitude, any attempt to generate a mapcode will fail.

Recall from Step 6 where, after constructing the address data, we passed the result along via an HTTP POST request in the createDigitalAddressApp.js file:

/var/www/html/digiaddress/js/createDigitalAddressApp.js

$http({
    method: 'POST',
    url: 'geoimplement.php',
    data: {address: fullAddress},
    headers: {'Content-Type': 'application/x-www-form-urlencoded'}
}).then(function successCallback(results) {

This code block sends the address data entered by a user to the geoimplement.php file which contains the code that calls the Google Maps API. Go ahead and open this file:

nano /var/www/html/digiaddress/geoimplement.php

You'll see that it first decodes the address that was received through the POST request:

/var/www/html/digiaddress/geoimplement.php

. . .
$data=json_decode(file_get_contents("php://input"));
. . .

It then passes the address field of the input data to a geocode function which returns the geographic information on the address:

/var/www/html/digiaddress/geoimplement.php

. . .
$result = geocode($data->address);
. . .

The result is then echoed back to the caller:

/var/www/html/digiaddress/geoimplement.php

. . .
echo json_encode($result);
. . .

The geocode function encodes the address and passes it on to the Google Maps API, along with your application key:

/var/www/html/digiaddress/geoimplement.php

. . .
// url encode the address
$address = urlencode($address);

// google map geocode api url
$url = "https://maps.googleapis.com/maps/api/geocode/json?address={$address}&key=<YOUR KEY>";
. . .

Before scrolling on, go ahead and add your API key to the line under the // google map geocode api url comment:

/var/www/html/digiaddress/geoimplement.php

. . .
// google map geocode api url
$url = "https://maps.googleapis.com/maps/api/geocode/json?address={$address}&key=ExampleAPIKeyH2vITfv1eIHbfka9ym634Esw7u";
. . .

After sending the call to the Google Maps API, the response is decoded and its value is returned by the function:

/var/www/html/digiaddress/geoimplement.php

. . .
// get the json response
$resp_json = file_get_contents($url);

// decode the json
$resp = json_decode($resp_json, true);

if ($resp['status'] == 'OK') {
    return $resp['results'][0];
} else {
    return false;
}
. . .

Save this file, and visit your application once again. Input US-NY in the state field and then hit TAB to change the input focus to the next field. You will see the following output:

Notice that the geocoordinates and physical address that you entered in the form appear underneath the map. This makes the application feel much more engaging and interactive.

Note: When it comes to abbreviations for place names, Mapcode uses the ISO 3166 standard. This means that it may not interpret some commonly-used abbreviations as expected. For example, if you'd like to generate a Mapcode for an address in Louisiana and you enter LA, the map will jump to Los Angeles, California (rather than the state of Louisiana).

You can avoid confusion with US postal abbreviations by preceding them with US-. In the context of this Louisiana example, you would enter US-LA.

To learn more about how Mapcode uses this standard, check out the Territories and standard codes reference page.

Despite this improvement to how the application displays locations on the map, the app still isn't fully functional. The last step you need to take before you can generate a mapcode is to edit the db.php file to allow the application to access your database.

Step 9 — Adding Database Credentials and Testing Mapcode Generation

Recall that this application stores every address entered into the form — along with its latitude, longitude, and mapcode — in the database you created in Step 2. This is made possible by the code within the db.php file, which stores your database credentials and allows the application to access the locations table within it.

As a final step to enable the mapcode generation functionality, open the db.php file for editing:

nano /var/www/html/digiaddress/db.php

Near the top of this file, find the line that begins with $pass. This line submits your MySQL login credentials in order to allow the application to access your database. Replace your_password with your root MySQL user's password:

/var/www/html/digiaddress/db.php

. . .
        $username = "root";
        $pass = "your_password";
. . .

That is the last change you need to make in order to generate a mapcode from a physical address. Save and close the file, then go ahead and refresh the application in your browser once again. Enter in an address of your choice and click the Generate button. The output will look similar to this:

At this stage, you have completed your application and you can now generate a short digital address for any physical location in the world. Feel free to experiment with different addresses, and note that the address you enter does not necessarily need to be within the United States.

Your final task is to enable this app's second functionality: retrieving an address from the database using its respective mapcode.

Step 10 — Retrieving a Physical Address

Now that you're able to generate a mapcode from a given physical address, your final step is to retrieve the original physical address, as derived from the mapcode. To accomplish this, we will develop a PHP user interface, shown here:

The code for this UI is available in the findaddress.php file. As the UI defined within this file is fairly similar to the UI we covered earlier in Step 4, we will not look too closely at all the details of how it works. We will, however, go through these three files to explain generally how they function.

In order to enable the address retrieval functionality, you'll need to add your Google API key to the findaddress.php file, so open it up with your preferred editor:

nano /var/www/html/digiaddress/findaddress.php

Near the bottom of the file, find the line that begins with <script async defer src=. It will look like this:

/var/www/html/digiaddress/findaddress.php

<script async defer src="https://maps.googleapis.com/maps/api/js?key=<YOUR KEY>"></script>

Replace <YOUR KEY> with your Google API key as you've done in the previous steps, then save the file. Before closing it, though, let's take a quick look to see how these files work together.

When a user submits the form it triggers a submit event, and an event listener calls the fetchadd function:

/var/www/html/digiaddress/findaddress.php

. . .
<form ng-submit="fetchadd()" class="custom-form">
. . .

The fetchadd function sends the digital address to fetchaddress.php with a POST request:

/var/www/html/digiaddress/js/findAddressApp.js

. . .
$http({
    method : 'POST',
    url : 'fetchaddress.php',
    data : {digiaddress: $scope.digiaddress}
}).then(function(response){
. . .

If the POST is successful, the function returns a JSON response. The following line parses this response:

/var/www/html/digiaddress/js/findAddressApp.js

. . .
var jsonlatlng = JSON.parse(response.data.latlng);
. . .

The next lines set the marker on the map:

/var/www/html/digiaddress/js/findAddressApp.js

. . .
marker = new google.maps.Marker({
    position: new google.maps.LatLng(jsonlatlng.latitude, jsonlatlng.longitude),
        map: locationMap
});
. . .

And the following prints the geocoordinates and the physical address:

/var/www/html/digiaddress/js/findAddressApp.js

. . .
geoCoordLabel = angular.element(document.querySelector('#geocoordinates'));
geoCoordLabel.html("Geo Coordinate: "+ jsonlatlng.latitude +","+ jsonlatlng.longitude);

geoAddressLabel = angular.element(document.querySelector('#geoaddress'));
geoAddressLabel.html("Geo Address: " + jsonlatlng.house +","+ jsonlatlng.town +","+ jsonlatlng.street +","+ jsonlatlng.state + " " + jsonlatlng.zip );
. . .

Visit this application in your browser by going to the following link:

http://your_server_ip/digiaddress/findaddress.php

Test it out by entering in the mapcode you obtained earlier. The following figure shows a typical output:

With that, your application is finished. You can now create a unique mapcode for any location in the world, and then use that mapcode to retrieve the location's physical address.

Conclusion

In this tutorial you used the Google Maps API to pin a location and gets its longitude, latitude information. This information is used to generate a unique and short digital address using Mapcode API. There are a number of practical use cases for mapcodes, ranging from emergency services to archaeological surveying. The Stichting Mapcode Foundation lists several such use cases.

Acknowledgements

Many thanks to Dinesh Karpe and Sayli Patil for developing the entire project code.

How to Retrieve Let's Encrypt SSL Wildcard Certificates using CloudFlare Validation on CentOS 7

2018-08-13T15:54:36Z

The author selected Code.org to receive a donation as part of the Write for DOnations program.

Introduction

Let's Encrypt is a certificate authority (CA) that provides free certificates for Transport Layer Security (TLS) encryption. It provides a software client called Certbot which simplifies the process of certificate creation, validation, signing, installation, and renewal.

Let’s Encrypt now supports wildcard certificates which allow you to secure all subdomains of a domain with a single certificate. This will be useful if you want to host multiple services, such as web interfaces, APIs, and other sites using a single server.

To obtain a wildcard certificate from Let’s Encrypt you have to use one of Certbot’s DNS plugins, which include:

certbot-dns-cloudflare
certbot-dns-route53
certbot-dns-google
certbot-dns-digitalocean

The plugin you choose depends on which service hosts your DNS records. In this tutorial you will obtain a wildcard certificate for your domain using CloudFlare validation with Certbot on CentOS 7. You'll then configure the certificate to renew it when it expires.

Prerequisites

To complete this tutorial, you'll need the following:

One CentOS 7 server set up by following the CentOS 7 initial server setup guide, including a sudo non-root user and a firewall.
A fully registered domain name. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.
A Cloudflare account.
A DNS record set up for your domain in Cloudflare's DNS, along with a couple of subdomains configured. You can follow CloudFlare's tutorial on setting up a web site to configure this.

Step 1 — Installing Certbot

The certbot package is not available through CentOS's package manager by default. You will need to enable the EPEL repository to install Certbot and its plugins.

To add the CentOS 7 EPEL repository, run the following command:

sudo yum install -y epel-release

Once the installation completes, you can install certbot:

sudo yum install -y certbot

And then install the CloudFlare plugin for Certbot:

sudo yum install -y python2-cloudflare python2-certbot-dns-cloudflare

If you are using another DNS service, you can find the corresponding plugin using the yum search command:

yum search python2-certbot-dns

You've prepared your server to obtain certificates. Now you need to get the API key from CloudFlare.

Step 2 — Getting the CloudFlare API

In order for Certbot to automatically renew wildcard certificates, you need to provide it with your CloudFlare login and API key.

Click the View button in the Global API Key line.

For security reasons, you will be asked to re-enter your Cloudflare account password. Enter it and validate the CAPTCHA. Then click the View button again. You'll see your API key:

Copy this key. You will use it in the next step.

Now return to your server to continue the process of obtaining the certificate.

Step 3 — Configuring Certbot

You have all of the necessary information to tell Certbot how to use Cloudflare, but let's write it to a configuration file so that Сertbot can use it automatically.

First run the certbot command without any parameters to create the initial configuration file:

sudo certbot

Next create a configuration file in the /etc/letsencrypt directory which will contain your CloudFlare email and API key:

sudo vi /etc/letsencrypt/cloudflareapi.cfg

Add the following into it, replacing the placeholders with your Cloudflare login and API key:

/etc/letsencrypt/cloudflareapi.cfg

dns_cloudflare_email = your_cloudflare_login
dns_cloudflare_api_key = your_cloudflare_api_key

Save the file and exit the editor.
With Cloudflare's API key, you can do the same things from the command line that you can do from the Cloudflare UI, so in order to protect your account, make the configuration file readable only by its owner so nobody else can obtain your key:

sudo chmod 600 /etc/letsencrypt/cloudflareapi.cfg

With the configuration files in place, let's obtain a certificate.

Step 4 — Obtaining the Certificate

To obtain a certificate, we'll use the certbot command and specify the plugin we want, the credentials file we want to use, and the server we should use to handle the request. By default, Certbot uses Let’s Encrypt’s production servers, which use ACME API version 1, but Certbot uses another protocol for obtaining wildcard certificates, so you need to provide an ACME v2 endpoint.

Run the following command to obtain the wildcard certificate for your domain:

sudo certbot certonly --cert-name your_domain --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflareapi.cfg --server https://acme-v02.api.letsencrypt.org/directory -d "*.your_domain" -d your_domain

You will be asked to specify the email address that should receive urgent renewal and security notices:

Output...
Plugins selected: Authenticator dns-cloudflare, Installer None
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): your email

Then you'll be asked to agree to the Terms of Service:

Output-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v02.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: A

Then you'll be asked to share your email address with the Electronic Frontier
Foundation:

Output-------------------------------------------------------------------------------
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about EFF and
our work to encrypt the web, protect its users and defend digital rights.
-------------------------------------------------------------------------------
(Y)es/(N)o: N

Then Certbot will obtain your certificates. You will see the following message:

OutputIMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/your_domain/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/your_domain/privkey.pem
   Your cert will expire on 2018-07-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Now you have your wildcard certificate. Let's take a look at what Certbot has downloaded for you. Use the ls command to see the contents of the directory that holds your keys and certificates:

sudo ls /etc/letsencrypt/live/your_domain

Outputcert.pem  chain.pem  fullchain.pem  privkey.pem  README

The README file contains information about these files:

$ cat /etc/letsencrypt/live/your_domain/README

You'll see output like this:

README

This directory contains your keys and certificates.

`privkey.pem`  : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem`     : will break many server configurations, and should not be used
                 without reading further documentation (see link below).

We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.

From here, you can configure your servers with the wildcard certificate. You'll usually only need two of these files: fullchain.pem and privkey.pem.

For example, you can configure several web-based services:

wwww.example.com
api.example.com
mail.example.com

To do this, you will need a web server, such as Apache or Nginx. The installation and configuration of these servers is beyond the scope of this tutorial, but the following guides will walk you through all the necessary steps to configure the servers and apply your certificates.

For Nginx, take a look at these tutorials:

For Apache, consult these tutorials:

Now let's look at renewing the certificates automatically.

Step 5 — Renewing certificates

Let’s Encrypt issues short-lived certificates which are valid for 90 days. We'll need to set up a cron task to check for expiring certificates and renew them automatically.

Let's create a cron task
which will run the renewal check daily.

Use the following command to open the crontab file for editing:

sudo crontab -e

Add the following line to the file to attempt to renew the certificates daily:

crontab

30 2 * * * certbot renew --noninteractive

30 2 * * * means "run the following command at 2:30 am, every day".
The certbot renew command will check all certificates installed on the system and update any that are set to expire in less than thirty days.
--noninteractive tells Certbot not to wait for user input.

You will need to reload your web server after updating your certificates. The renew command includes hooks for running commands or scripts before or after a certificate is renewed. You can also configure these hooks in the renewal configuration file for your domain.

For example, to reload your Nginx server, open the renewal configuration file:

sudo vi /etc/letsencrypt/renewal/your_domain.conf

Then add the following line under the [renewalparams] section:

your_domain.conf'>/etc/letsencrypt/renewal/your_domain.conf

renew_hook = systemctl reload nginx

Now Certbot will automatically restart your web server after installing the updated certificate.

Conclusion

In this tutorial you've installed the Certbot client, obtained your wildcard certificate using DNS validation and enabled automatic renewals. This will allow you to use a single certificate with multiple subdomains of your domain and secure your web services.

Como Fazer Scraping em Páginas Web com Beautiful Soup and Python 3

2018-07-09T21:46:48Z

Introdução

Muitos projetos de análise de dados, big data, e aprendizado de máquina exigem o scraping de websites para coletar os dados com os quais você irá trabalhar. A linguagem de programação Python é largamente utilizada na comunidade de data science, e, portanto, tem um ecossistema de módulos e ferramentas que você pode usar em seus próprios projetos. Neste tutorial estaremos nos concentrando no módulo Beautiful Soup.

Beautiful Soup, uma alusão à música Mock Turtle’s encontrada no Capítulo 10 de Alice no País das Maravilhas, de Lewis Carroll, é uma biblioteca do Python que permite um retorno rápido em projetos de web scraping. Atualmente disponível como Beautiful Soup 4 e compatível tanto com Python 2.7 quanto com Python 3, o Beautiful Soup cria uma árvore de análise a partir de documentos HTML e XML analisados (incluindo documentos com tags não fechadas ou tag soup e outras marcações malformadas).

Neste tutorial, iremos coletar e analisar uma página web de forma a pegar dados textuais e gravar as informações que tivermos recolhido em um arquivo CSV.

Pré-requisitos

Antes de trabalhar com este tutorial, você deve ter um ambiente de programação Python local ou baseado em servidor configurado em sua máquina.

Você deve ter os módulos Requests e Beautiful Soup instalados, o que pode ser conseguido seguindo o nosso tutorial “How To Work with Web Data Using Requests and Beautiful Soup with Python 3.” Também seria útil ter familiaridade no trabalho com esses módulos.

Adicionalmente, uma vez que vamos trabalhar com dados extraídos da web, você deve estar confortável com a estrutura e a marcação de tags HTML.

Entendendo os Dados

Neste tutorial, iremos trabalhar com dados do site oficial do National Gallery of Art nos Estados Unidos. O National Gallery é um museu de arte localizado no National Mall em Washington, D.C. Ele possui mais de 120.000 peças datadas desde o Renascimento aos dias atuais feitas por mais de 13.000 artistas.

Gostaríamos de pesquisar o Índice de Artistas, que, no momento da atualização deste tutorial, estava disponível via Internet Archive’s Wayback Machine na seguinte URL:

https://web.archive.org/web/20170131230332/https://www.nga.gov/collection/an.shtm

Nota: A longa URL acima é devido a este site ter sido arquivado pelo Internet Archive.

O Internet Archive é uma biblioteca digital sem fins lucrativos que fornece acesso livre a sites da internet e outras mídias digitais. A organização tira instantâneos de websites para preservar a história dos sites, e atualmente, podemos acessar uma versão mais antiga do site da National Gallery que estava disponível quando este tutorial foi escrito pela primeira vez. O Internet Archive é uma boa ferramenta para se ter em mente sempre que estiver fazendo qualquer tipo de scraping de dados históricos, incluindo a comparação entre iterações do mesmo site e dados disponíveis.

Logo abaixo do cabeçalho do Internet Archive, você verá uma página como esta:

Como estamos fazendo esse projeto para aprender sobre o web scraping com o Beautiful Soup, não precisamos extrair muitos dados do site, por isso, vamos limitar o escopo dos dados do artista que estamos tentando capturar. Vamos, portanto, escolher uma letra — em nosso exemplo escolheremos a letra Z — e veremos uma página como esta:

Na página acima, vemos que o primeiro artista listado no momento da escrita é Zabaglia, Niccola, o que é uma boa coisa a se notar quando começamos a extrair dados. Começaremos trabalhando com essa primeira página, com a seguinte URL para a letra Z:

https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm

É importante observar, para análise posterior, quantas páginas existem para a letra que você está escolhendo listar, o que você pode descobrir clicando na última página de artistas. Nesse caso, existe um total de 4 páginas, e o último artista listado no momento da escrita é Zykmund, Václav. A última página de artistas com Z tem a seguinte URL:

https://web.archive.org/web/20121010201041/http://www.nga.gov/collection/anZ4.htm

Contudo, você também pode acessar a página acima usando a mesma string numérica do Internet Archive da primeira página:

https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ4.htm

É importante observar isso, pois mais adiante neste tutorial faremos a iteração dessas páginas.

Para começar a se familiarizar com a forma que essa página web é configurada, você pode dar uma olhada em seu DOM, que o ajudará a entender como o HTML é estruturado. Para inspecionar o DOM, você pode abrir Ferramentas do Desenvolvedor do seu navegador.

Importando as Bibliotecas

Para iniciar nosso projeto de codificação, vamos ativar nosso ambiente de programação. Certifique-se de que você está no diretório onde o seu ambiente de desenvolvimento está localizado, e execute o seguinte comando.

. my_env/bin/activate

Com o nosso ambiente de programação ativado, vamos criar um novo arquivo, com o nano por exemplo. Você pode nomear seu arquivo como quiser, vamos chamá-lo de nga_z_artists.py nesse tutorial.

nano nga_z_artists.py

Nesse arquivo, podemos começar a importar as bibliotecas que iremos utilizar — Requests e Beautiful Soup.

A biblioteca Requests lhe permite fazer uso do HTTP dentro dos seus programas Python em um formato legível, e o módulo Beautiful Soup é projetado para fazer web scraping rapidamente.

Vamos importar tanto o Requests quanto o Beautiful Soup com a declaração import. Para o Beautiful Soup iremos importá-lo do bs4, o pacote no qual o Beautiful Soup 4 é encontrado.

nga_z_artists.py


# Importar bibliotecas
import requests
from bs4 import BeautifulSoup

Com os módulos Requests e Beautiful Soup importados, podemos passar a trabalhar para coletar primeiro uma página e analisá-la.

Coletando e Analisando uma Página Web

O próximo passo que precisaremos fazer é coletar a URL da primeira página web com o Requests. Iremos atribuir a URL da primeira página à variável page usando o método requests.get().

nga_z_artists.py


import requests
from bs4 import BeautifulSoup


# Coletar a primeira página da lista de artistas
page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')

Nota: Como a URL é longa, o código acima, bem como todo este tutorial não passarão no PEP 8 E501, que sinaliza linhas com mais de 79 caracteres. Você pode querer atribuir a URL a uma variável para tornar o código mais legível nas versões finais. O código neste tutorial é para fins de demonstração e permitirá que você troque URLs mais curtas como parte de seus próprios projetos.

Agora iremos criar o objeto BeautifulSoup, ou uma árvore de análise. Esse objeto utiliza como argumento o documento page.text do Requests (o conteúdo da resposta do servidor) e então o analisa através do html.parser interno do Python.

nga_z_artists.py


import requests
from bs4 import BeautifulSoup


page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')

# Criar o objeto BeautifulSoup
soup = BeautifulSoup(page.text, 'html.parser')

Com a nossa página coletada, analisada e configurada como um objeto BeautifulSoup, podemos passar para a coleta dos dados que gostaríamos.

Pegando Texto de uma Página Web

Para este projeto, iremos coletar nomes de artistas e os links relevantes disponíveis no website. Você pode querer coletar dados diferentes, tais como a nacionalidade dos artistas e datas. Para quaisquer dados que você queira coletar, você precisa descobrir como ele é descrito pelo DOM da página web.

Para fazer isso, no seu navegador web, clique com o botão direito — ou CTRL + clique no macOS — no nome do primeiro artista, Zabaglia, Niccola. Dentro do menu de contexto que aparece, você deve ver um item de menu semelhante ao Inspecionar Elemento (Firefox) ou Inspecionar (Chrome).

Após clicar no item de menu relevante Inspecionar, as ferramentas para desenvolvedores web devem aparecer no seu navegador. Queremos procurar pela classe e as tags associadas aos nomes dos artistas nessa lista.

Veremos primeiro que a tabela de nomes está dentro de tags <div> onde class="BodyText". É importante observar isso, para que só procuremos texto nessa seção da página web. Também notamos que o nome Zabaglia, Niccola está em uma tag de link, já que o nome faz referência a uma página web que descreve o artista. Então, vamos querer referenciar a tag <a> para links. O nome de cada artista é uma referência a um link.

Para fazer isso, iremos utilizar os métodos find() e find_all() do Beautiful Soup a fim de extrair o texto dos nomes dos artistas do BodyText <div>.

nga_z_artists.py


import requests
from bs4 import BeautifulSoup


# Coletar e analisar a primeira página
page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
soup = BeautifulSoup(page.text, 'html.parser')

# Pegar todo o texto da div BodyText
artist_name_list = soup.find(class_='BodyText')

# Pegar o texto de todas as instâncias da tag <a> dentro da div BodyText
artist_name_list_items = artist_name_list.find_all('a')

A seguir, na parte inferior do nosso arquivo de programa, criaremos um loop for para iterar todos os nomes de artistas que acabamos de colocar na variável artist_name_list_items.

Vamos imprimir esses nomes com o método prettify() para transformar a árvore de análise do Beautiful Soup em uma string Unicode bem formatada.

nga_z_artists.py


...
artist_name_list = soup.find(class_='BodyText')
artist_name_list_items = artist_name_list.find_all('a')

# Criar loop para imprimir todos os nomes de artistas
for artist_name in artist_name_list_items:
    print(artist_name.prettify())

Vamos executar o programa como ele está até agora:

python nga_z_artists.py

Assim que fizermos isso, receberemos a seguinte saída:

Output<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630">
 Zabaglia, Niccola
</a>
...
<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3427">
 Zao Wou-Ki
</a>
<a href="/web/20121007172955/https://www.nga.gov/collection/anZ2.htm">
 Zas-Zie
</a>

<a href="/web/20121007172955/https://www.nga.gov/collection/anZ3.htm">
 Zie-Zor
</a>

<a href="/web/20121007172955/https://www.nga.gov/collection/anZ4.htm">
 <strong>
  next
  <br/>
  page
 </strong>
</a>

O que vemos na saída nesse ponto é o texto completo e as tags relativas a todos os nomes de artistas dentro de tags <a> encontradas na tag <div class="BodyText"> na primeira página, bem como algum texto de link adicional na parte inferior. Como não queremos essa informação extra, vamos trabalhar para remover isso na próxima seção.

Removendo Dados Supérfluos

Até agora, conseguimos coletar todos os dados de texto do link dentro de uma seção <div> da nossa página web. No entanto, não queremos ter os links inferiores que não fazem referência aos nomes dos artistas. Por isso, vamos trabalhar para remover essa parte.

Para remover os links inferiores da página, vamos clicar novamente com o botão direito e Inspecionar o DOM. Veremos que os links na parte inferior da seção <div class="BodyText"> estão contidos em uma tabela HTML: <table class="AlphaNav">:

Podemos, portanto, usar o Beautiful Soup para encontrar a classe AlphaNav e usar o método decompose() para remover uma tag da árvore de análise e depois destruí-la juntamente com seu conteúdo.

Usaremos a variável last_links para fazer referência a esses links inferiores e adicioná-los ao arquivo do programa:

nga_z_artists.py


import requests
from bs4 import BeautifulSoup


page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')

soup = BeautifulSoup(page.text, 'html.parser')

# Remover links inferiores
last_links = soup.find(class_='AlphaNav')
last_links.decompose()

artist_name_list = soup.find(class_='BodyText')
artist_name_list_items = artist_name_list.find_all('a')

for artist_name in artist_name_list_items:
    print(artist_name.prettify())

Agora, quando executarmos o programa com o comando python nga_z_artist.py, receberemos a seguinte saída:

Output<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630">
 Zabaglia, Niccola
</a>
<a href="/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202">
 Zaccone, Fabian
</a>
...
<a href="/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=11631">
 Zanotti, Giampietro
</a>
<a href="/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=3427">
 Zao Wou-Ki
</a>

Nesse ponto, vemos que a saída não inclui mais os links na parte inferior da página web e agora exibe apenas os links associados aos nomes dos artistas.

Até agora, focamos especificamente os links com os nomes dos artistas, mas temos os dados de tags extras que realmente não queremos. Vamos remover isso na próxima seção.

Pegando o Conteúdo de uma Tag

Para acessar apenas os nomes reais dos artistas, queremos focar no conteúdo das tags <a> em vez de imprimir toda a tag de link.

Podemos fazer isso com o .contents do Beautiful Soup, que irá retornar a tag filha com um tipo de dados lista do Python.

Vamos revisar o loop for para que, em vez de imprimir o link inteiro e sua tag, façamos a impressão da lista das filhas (ou seja, os nomes completos dos artistas).

nga_z_artists.py


import requests
from bs4 import BeautifulSoup


page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')

soup = BeautifulSoup(page.text, 'html.parser')

last_links = soup.find(class_='AlphaNav')
last_links.decompose()

artist_name_list = soup.find(class_='BodyText')
artist_name_list_items = artist_name_list.find_all('a')

# Usar .contents para pegar as tags <a> filhas
for artist_name in artist_name_list_items:
    names = artist_name.contents[0]
    print(names)

Note que estamos iterando na lista acima chamando o número do índice de cada item.

Podemos executar o programa com o comando python para ver a seguinte saída:

OutputZabaglia, Niccola
Zaccone, Fabian
Zadkine, Ossip
...
Zanini-Viola, Giuseppe
Zanotti, Giampietro
Zao Wou-Ki

Recebemos de volta uma lista de todos os nomes dos artistas disponíveis na primeira página da letra Z.

Mas, e se quisermos também capturar as URLs associadas a esses artistas? Podemos extrair URLs encontradas dentro de tags <a> utilizando o método get('href') do Beautiful Soup.

A partir da saída dos links acima, sabemos que a URL inteira não está sendo capturada, então vamos concatenar a string do link com o início da string da URL (nesse caso https://web.archive.org/).

Estas linhas também serão adicionadas ao loop for:

nga_z_artists.py


...
for artist_name in artist_name_list_items:
    names = artist_name.contents[0]
    links = 'https://web.archive.org' + artist_name.get('href')
    print(names)
    print(links)

Quando executamos o programa acima, receberemos tanto os nomes dos artistas quanto as URLs para os links que nos dizem mais sobre eles:

OutputZabaglia, Niccola
https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630
Zaccone, Fabian
https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202
...
Zanotti, Giampietro
https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11631
Zao Wou-Ki
https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3427

Embora estejamos agora recebendo informações do site, ele está apenas imprimindo em nossa janela de terminal. Em vez disso, vamos capturar esses dados para podermos usá-los em outro lugar, gravando-os em um arquivo.

Gravando os Dados em um Arquivo CSV

Coletar dados que só residem em uma janela de terminal não é muito útil. Arquivos de valores separados por vírgulas (CSV) nos permitem armazenar dados tabulares em texto plano, e é um formato comum para planilhas e bancos de dados. Antes de iniciar esta seção, você deve familiarizar-se com como manipular arquivos de texto sem formatação em Python.

Primeiro, precisamos importar o módulo interno csv do Python junto com os outros módulos no topo do arquivo de código:

import csv

Em seguida, vamos criar e abrir um arquivo chamado z-artist-names.csv para que possamos gravar nele (iremos utilizar aqui a variável f para o arquivo), utilizando o modo 'w'. Também vamos escrever os cabeçalhos da primeira linha: Name and Link que iremos passar para o método writerow() como uma lista.

f = csv.writer(open('z-artist-names.csv', 'w'))
f.writerow(['Name', 'Link'])

Finalmente, dentro do nosso loop for, vamos escrever cada linha com os names ou nomes dos artistas e seus links associados:

f.writerow([names, links])

Você pode ver as linhas para cada uma dessas tarefas no arquivo abaixo:

nga_z_artists.py


import requests
import csv
from bs4 import BeautifulSoup


page = requests.get('https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm')

soup = BeautifulSoup(page.text, 'html.parser')

last_links = soup.find(class_='AlphaNav')
last_links.decompose()

# Criar um arquivo para gravar, adicionar linha de cabeçalhos
f = csv.writer(open('z-artist-names.csv', 'w'))
f.writerow(['Name', 'Link'])

artist_name_list = soup.find(class_='BodyText')
artist_name_list_items = artist_name_list.find_all('a')

for artist_name in artist_name_list_items:
    names = artist_name.contents[0]
    links = 'https://web.archive.org' + artist_name.get('href')


    # Adicionar em uma linha o nome de cada artista e o link associado
    f.writerow([names, links])

Quando você executar o programa agora com o comando python, nenhuma saída será retornada para sua janela de terminal. Em vez disso, um arquivo será criado no diretório em que você está trabalhando, chamado z-artist-names.csv.

Dependendo do que você usa para abrí-lo, ele deve ser algo assim:

z-artist-names.csv


Name,Link
"Zabaglia, Niccola",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=11630
"Zaccone, Fabian",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=34202
"Zadkine, Ossip",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=3475w
...

Ou, ele pode se parecer mais com uma planilha:

Em ambos os casos, agora você pode usar esse arquivo para trabalhar com os dados de maneiras mais significativas, já que as informações coletadas agora estão armazenadas no disco do seu computador.

Recuperando Páginas Relacionadas

Criamos um programa que extrairá dados da primeira página da lista de artistas cujos sobrenomes começam com a letra Z. Porém, existem 4 páginas desses artistas no total, disponíveis no website.

Para coletar todas essas páginas, podemos executar mais iterações com loops for. Isso revisará a maior parte do código que escrevemos até agora, mas empregará conceitos semelhantes.

Para começar, vamos inicializar uma lista para manter as páginas:

pages = []

Vamos preencher essa lista inicializada com o seguinte loop for:

for i in range(1, 5):
    url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
    pages.append(url)

Anteriormente neste tutorial, observamos que devemos prestar atenção ao número total de páginas que contêm nomes de artistas começando com a letra Z (ou qualquer letra que estivermos utilizando). Uma vez que existem 4 páginas para a letra Z, construímos o loop foracima com um intervalo de 1 a 5 de modo que ele vai iterar através de cada uma das 4 páginas.

Para este website específico, as URLs começam com a string https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZe são seguidas com um número de página (que será o inteiro i do loop for que convertemos para uma string) e terminam com .htm. Iremos concatenar estas strings e depois acrescentar o resultado à lista pages.

Além desse loop, teremos um segundo loop que passará por cada uma das páginas acima. O código nesse loop for será parecido com o código que criamos até agora, já que ele está executando a tarefa que completamos para a primeira página dos artistas com a letra Z para cada um das 4 páginas do total. Observe que, como colocamos o programa original no segundo loop for, agora temos o loop original como um loop for aninhado contido nele.

Os dois loops for ficarão assim:

pages = []

for i in range(1, 5):
    url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
    pages.append(url)

for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

    last_links = soup.find(class_='AlphaNav')
    last_links.decompose()

    artist_name_list = soup.find(class_='BodyText')
    artist_name_list_items = artist_name_list.find_all('a')

    for artist_name in artist_name_list_items:
        names = artist_name.contents[0]
        links = 'https://web.archive.org' + artist_name.get('href')

        f.writerow([names, links])

No código acima, você deve ver que o primeiro loop for está iterando nas páginas e o segundo loop for está extraindo dados de cada uma dessas páginas e, em seguida, adicionando os nomes e links dos artistas, linha por linha, em cada linha de cada página.

Estes dois loops for estão abaixo das declaraçõs import, da criação e escrita do arquivo CSV (com a linha para a escrita dos cabeçalhos do arquivo), e a inicialização da variável pages (atribuída a uma lista).

Dentro de um contexto macro do arquivo de programação, o código completo se parece com isto:

nga_z_artists.py


import requests
import csv
from bs4 import BeautifulSoup


f = csv.writer(open('z-artist-names.csv', 'w'))
f.writerow(['Name', 'Link'])

pages = []

for i in range(1, 5):
    url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
    pages.append(url)


for item in pages:
    page = requests.get(item)
    soup = BeautifulSoup(page.text, 'html.parser')

    last_links = soup.find(class_='AlphaNav')
    last_links.decompose()

    artist_name_list = soup.find(class_='BodyText')
    artist_name_list_items = artist_name_list.find_all('a')

    for artist_name in artist_name_list_items:
        names = artist_name.contents[0]
        links = 'https://web.archive.org' + artist_name.get('href')

        f.writerow([names, links])

Como esse programa está fazendo um trabalho, levará algum tempo para criar o arquivo CSV. Depois de concluído, a saída estará comleta, mostrando os nomes dos artistas e seus links associados de Zabaglia, Niccola até Zykmund, Václav.

Sendo Cuidadoso

Ao fazer scraping em páginas web, é importante manter-se cuidadoso com os servidores dos quais você está pegando informações.

Verifique se o site tem termos de serviço ou termos de uso relacionados ao web scraping. Além disso, verifique se o site tem uma API que permite coletar dados antes de você mesmo fazer scraping.

Certifique-se de não acessar continuamente os servidores para coletar dados. Depois de coletar o que você precisa de um site, execute scripts que vasculhem pelos dados localmente, em vez de sobrecarregar os servidores de outra pessoa.

Adicionalmente, é uma boa ideia fazer web scraping com um cabeçalho que tenha o seu nome e e-mail para que o website possa identificá-lo e fazer o acompanhamento caso tenha alguma dúvida. Um exemplo de cabeçalho que você pode usar com a biblioteca Requests do Python é o seguinte:

import requests

headers = {
    'User-Agent': 'Seu nome, example.com',
    'From': 'email@example.com'
}

url = 'https://example.com'

page = requests.get(url, headers = headers)

A utilização de cabeçalhos com informações identificáveis garante que as pessoas que acessam os logs de um servidor possam entrar em contato com você.

Conclusão

Este tutorial usou o Python e o Beautiful Soup para coletar dados de um website. Armazenamos o texto que reunimos em um arquivo CSV.

Você pode continuar trabalhando neste projeto coletando mais dados e tornando seu arquivo CSV mais robusto. Por exemplo, você pode querer incluir as nacionalidades e os anos de cada artista. Você também pode usar o que aprendeu para coletar dados de outros sites.

Para continuar aprendendo sobre como extrair informações da web, leia nosso tutorial “How To Crawl A Web Page with Scrapy and Python 3.”

How To Install Software on Kubernetes Clusters with the Helm Package Manager

2018-08-09T18:30:35Z

Introduction

Helm is a package manager for Kubernetes that allows developers and operators to more easily configure and deploy applications on Kubernetes clusters.

In this tutorial we will set up Helm and use it to install, reconfigure, rollback, then delete an instance of the Kubernetes Dashboard application. The dashboard is an official web-based Kubernetes GUI.

For a conceptual overview of Helm and its packaging ecosystem, please read our article An Introduction to Helm.

Prerequisites

For this tutorial you will need:

A Kubernetes 1.8+ cluster with role-based access control (RBAC) enabled.
The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

You can test your connectivity with the following command:
```
kubectl cluster-info
```
If you see no errors, you're connected to the cluster. If you access multiple clusters with kubectl, be sure to verify that you've selected the correct cluster context:
```
kubectl config get-contexts
```
```
OutputCURRENT   NAME                    CLUSTER                      AUTHINFO                      NAMESPACE
*         do-nyc1-k8s-example     do-nyc1-k8s-example          do-nyc1-k8s-example-admin
          docker-for-desktop      docker-for-desktop-cluster   docker-for-desktop
```
In this example the asterisk (*) indicates that we are connected to the do-nyc1-k8s-example cluster. To switch clusters run:
```
kubectl config use-context context-name
```

When you are connected to the correct cluster, continue to Step 1 to begin installing Helm.

Step 1 — Installing Helm

First we'll install the helm command-line utility on our local machine. Helm provides a script that handles the installation process on MacOS, Windows, or Linux.

Change to a writable directory and download the script from Helm's GitHub repository:

cd /tmp
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh

Make the script executable with chmod:

chmod u+x install-helm.sh

At this point you can use your favorite text editor to open the script and inspect it to make sure it’s safe. When you are satisfied, run it:

./install-helm.sh

You may be prompted for your password. Provide it and press ENTER.

Outputhelm installed into /usr/local/bin/helm
Run 'helm init' to configure helm.

Next we will finish the installation by installing some Helm components on our cluster.

Step 2 — Installing Tiller

Tiller is a companion to the helm command that runs on your cluster, receiving commands from helm and communicating directly with the Kubernetes API to do the actual work of creating and deleting resources. To give Tiller the permissions it needs to run on the cluster, we are going to make a Kubernetes serviceaccount resource.

Note: We will bind this serviceaccount to the cluster-admin cluster role. This will give the tiller service superuser access to the cluster and allow it to install all resource types in all namespaces. This is fine for exploring Helm, but you may want a more locked-down configuration for a production Kubernetes cluster.

Please refer to the official Helm RBAC documentation for more information on setting up different RBAC scenarios for Tiller.

Create the tiller serviceaccount:

kubectl -n kube-system create serviceaccount tiller

Next, bind the tiller serviceaccount to the cluster-admin role:

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

Now we can run helm init, which installs Tiller on our cluster, along with some local housekeeping tasks such as downloading the stable repo details:

helm init --service-account tiller

Output. . .

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

To verify that Tiller is running, list the pods in thekube-system namespace:

kubectl get pods --namespace kube-system

OutputNAME                                    READY     STATUS    RESTARTS   AGE
. . .
kube-dns-64f766c69c-rm9tz               3/3       Running   0          22m
kube-proxy-worker-5884                  1/1       Running   1          21m
kube-proxy-worker-5885                  1/1       Running   1          21m
kubernetes-dashboard-7dd4fc69c8-c4gwk   1/1       Running   0          22m
tiller-deploy-5c688d5f9b-lccsk          1/1       Running   0          40s

The Tiller pod name begins with the prefix tiller-deploy-.

Now that we've installed both Helm components, we're ready to use helm to install our first application.

Step 3 — Installing a Helm Chart

Helm software packages are called charts. Helm comes preconfigured with a curated chart repository called stable. You can browse the available charts in their GitHub repo. We are going to install the Kubernetes Dashboard as an example.

Use helm to install the kubernetes-dashboard package from the stable repo:

helm install stable/kubernetes-dashboard --name dashboard-demo

Output
NAME:   dashboard-demo
LAST DEPLOYED: Wed Aug  8 20:11:07 2018
NAMESPACE: default
STATUS: DEPLOYED

. . .

Notice the NAME line, highlighted in the above example output. In this case we specified the name dashboard-demo. This is the name of our release. A Helm release is a single deployment of one chart with a specific configuration. You can deploy multiple releases of the same chart with, each with its own configuration.

If you don't specify your own release name using --name, Helm will create a random name for you.

We can ask Helm for a list of releases on this cluster:

helm list

OutputNAME            REVISION    UPDATED                     STATUS      CHART                       NAMESPACE
dashboard-demo    1           Wed Aug  8 20:11:11 2018    DEPLOYED    kubernetes-dashboard-0.7.1  default

We can now use kubectl to verify that a new service has been deployed on the cluster:

kubectl get services

OutputNAME                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
dashboard-demo-kubernetes-dashboard   ClusterIP   10.32.104.73   <none>        443/TCP   51s
kubernetes                             ClusterIP   10.32.0.1      <none>        443/TCP   34m

Notice that by default the service name corresponding to our release is a combination of the Helm release name and the chart name.

Now that we've deployed the application, let's use Helm to change its configuration and update the deployment.

Step 4 — Updating a Release

The helm upgrade command can be used to upgrade a release with a new or updated chart, or update the it’s configuration options.

We're going to make a simple change to our dashboard-demo release to demonstrate the update and rollback process: we'll update the name of the dashboard service to just dashboard, instead of dashboard-demo-kubernetes-dashboard.

The kubernetes-dashboard chart provides a fullnameOverride configuration option to control the service name. Let's run helm upgrade with this option set:

helm upgrade dashboard-demo stable/kubernetes-dashboard --set fullnameOverride="dashboard"

You'll see output similar to the initial helm install step.

Check if your Kubernetes services reflect the updated values:

kubectl get services

OutputNAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes             ClusterIP   10.32.0.1       <none>        443/TCP   36m
dashboard              ClusterIP   10.32.198.148   <none>        443/TCP   40s

Our service name has been updated to the new value.

Note: At this point you may want to actually load the Kubernetes Dashboard in your browser and check it out. To do so, first run the following command:

kubectl proxy

This creates a proxy that lets you access remote cluster resources from your local computer. Based on the previous instructions your dashboard service is named kubernetes-dashboard and it's running in the default namespace. You may now access the dashboard at the following url:

http://localhost:8001/api/v1/namespaces/default/services/https:dashboard:/proxy/

If necessary, substitute your own service name and namespace for the highlighted portions. Instructions for actually using the dashboard are out of scope for this tutorial, but you can read the official Kubernetes Dashboard docs for more information.

Next we'll look at Helm's ability to roll back releases.

Step 5 — Rolling Back a Release

When we updated our dashboard-demo release in the previous step, we created a second revision of the release. Helm retains all the details of previous releases in case you need to roll back to a prior configuration or chart.

Use helm list to inspect the release again:

helm list

OutputNAME            REVISION    UPDATED                     STATUS      CHART                       NAMESPACE
dashboard-demo  2         Wed Aug  8 20:13:15 2018    DEPLOYED    kubernetes-dashboard-0.7.1  default

The REVISION column tells us that this is now the second revision.

Use helm rollback to roll back to the first revision:

helm rollback dashboard-demo 1

You should see the following output, indicating that the rollback succeeded:

OutputRollback was a success! Happy Helming!

At this point, if you run kubectl get services again, you will notice that the service name has changed back to its previous value. Helm has re-deployed the application with revision 1's configuration.

Next we'll look into deleting releases with Helm.

Step 6 — Deleting a Release

Helm releases can be deleted with the helm delete command:

helm delete dashboard-demo

Outputrelease "dashboard-demo" deleted

Though the release has been deleted and the dashboard application is no longer running, Helm saves all the revision information in case you want to re-deploy the release. If you tried to helm install a new dashboard-demo release right now, you'd get an error:

Error: a release named dashboard-demo already exists.

If you use the --deleted flag to list your deleted releases, you'll see that the release is still around:

helm list --deleted

OutputNAME            REVISION    UPDATED                     STATUS  CHART                       NAMESPACE
dashboard-demo  3           Wed Aug  8 20:15:21 2018    DELETED kubernetes-dashboard-0.7.1  default

To really delete the release and purge all old revisions, use the --purge flag with the helm delete command:

helm delete dashboard-demo --purge

Now the release has been truly deleted, and you can reuse the release name.

Conclusion

In this tutorial we installed the helm command-line tool and its tiller companion service. We also explored installing, upgrading, rolling back, and deleting Helm charts and releases.

For more information about Helm and Helm charts, please see the official Helm documentation.

Arquitetando Aplicações para o Kubernetes

2018-08-15T18:44:13Z

Introdução

Projetar e executar aplicações com escalabilidade, portabilidade e robustez pode ser um desafio, especialmente à medida que a complexidade do sistema aumenta. A arquitetura de uma aplicação ou sistema impacta significativamente em como ele deve ser executado, o que ele espera de seu ambiente e o quanto está acoplado aos componentes relacionados. Seguir certos padrões durante a fase de projeto e aderir a certas práticas operacionais pode ajudar a combater alguns dos problemas mais comuns que as aplicações enfrentam ao executar em ambientes altamente distribuídos.

Embora os padrões de projeto de software e as metodologias de desenvolvimento possam produzir aplicações com as características corretas de escalabilidade, a infraestrutura e o ambiente influenciam a operação do sistema implantado. Tecnologias como o Docker e o Kubernetes ajudam equipes a empacotar software e depois distribuir, fazer o deploy e escalar em plataformas de computadores distribuídos. Aprender a aproveitar melhor o poder dessas ferramentas pode ajudá-lo a gerenciar aplicações com maior flexibilidade, controle e capacidade de resposta.

Neste guia, vamos discutir alguns dos princípios e padrões que você pode adotar para ajudá-lo a dimensionar e gerenciar suas cargas de trabalho no Kubernetes. Embora o Kubernetes possa executar muitos tipos de cargas de trabalho, as escolhas feitas podem afetar a facilidade de operação e as possibilidades disponíveis no deploy. A maneira como você projeta e constrói suas aplicações, empacota seus serviços em containers e configura o gerenciamento do ciclo de vida e o comportamento dentro do Kubernetes pode, cada uma, influenciar sua experiência.

Projetando para Escalabilidade das Aplicações

Ao produzir software, muitos requisitos afetam os padrões e a arquitetura que você escolhe empregar. Com o Kubernetes, um dos fatores mais importantes é a capacidade de escalar horizontalmente, ajustando o número de cópias idênticas da sua aplicação para distribuir a carga e aumentar a disponibilidade. Esta é uma alternativa à escalabilidade vertical, que tenta manipular os mesmos fatores através do deploy em máquinas com maiores ou menores recursos.

Em particular, microservices ou micro-serviços é um padrão de projeto de software que funciona bem para deployments escaláveis em clusters. Os desenvolvedores criam aplicações pequenas e compostas que se comunicam pela rede através de APIs REST bem definidas, em vez de grandes programas compostos que se comunicam através de mecanismos internos de programação. A decomposição de aplicações monolíticas em componentes discretos de propósito único torna possível dimensionar cada função de forma independente. Grande parte da complexidade e composição que normalmente existiria no nível da aplicação é transferida para o domínio operacional, onde pode ser gerenciada por plataformas como o Kubernetes.

Além dos padrões de software específicos, as aplicações cloud native ou nativas para nuvem são projetadas com algumas considerações adicionais em mente. Aplicações cloud native são programas que seguem um padrão de arquitetura de microservices com resiliência integrada, observabilidade e recursos administrativos para se adaptar ao ambiente fornecido por plataformas em cluster na nuvem.

Por exemplo, aplicações cloud native são construídas com métricas de relatórios de integridade para permitir que a plataforma gerencie eventos do ciclo de vida se uma instância se tornar inconsistente. Eles produzem (e tornam disponível para exportação) dados robustos de telemetria para alertar os operadores sobre problemas e permitir que eles tomem decisões esclarecidas. As aplicações são projetadas para lidar com reinicializações e falhas regulares, alterações na disponibilidade do back-end e alta carga, sem corromper os dados ou deixar de responder.

Seguindo a Filosofia 12 Factor Application

Uma metodologia popular que pode ajudá-lo a se concentrar nas características mais importantes ao criar aplicações web prontas para a nuvem é a filosofia Twelve-Factor App. Escritos para ajudar os desenvolvedores e as equipes de operações a entender as principais qualidades compartilhadas pelos web services projetados para serem executados na nuvem, os princípios se aplicam muito bem ao software que funcionará em um ambiente em cluster como o Kubernetes. Embora os aplicativos monolíticos possam se beneficiar seguindo estas recomendações, as arquiteturas de microservices projetadas em torno desses princípios funcionam particularmente bem.

Um breve sumário dos Doze Fatores são:

Base de código: Gerencie todo o código em sistemas de controle de versão (como Git ou Mercurial). A base de código determina de forma abrangente o que é implantado.
Dependências: As dependências devem ser gerenciadas completa e explicitamente pela base de código, seja armazenada com o código ou com a versão fixada em um formato no qual um gerenciador de pacotes possa instalar.
Configuração: Separe os parâmetros de configuração da aplicação e defina-os no ambiente de deployment, em vez de mantê-los dentro da própria aplicação.
Serviços de apoio: Os serviços locais e remotos são abstraídos como recursos acessíveis pela rede, com detalhes de conexão definidos na configuração.
Construa, libere, execute: O estágio de construção da sua aplicação deve ser completamente separado dos processos de liberação e operação da mesma. O estágio de construção cria um artefato de deployment a partir do código-fonte, o estágio de liberação combina o artefato e a configuração, e o estágio de execução executa o release.
Processos: Aplicações são implementadas como processos que não devem contar com o armazenamento de estado localmente. O estado deve ser transferido para um serviço de apoio, conforme descrito no quarto fator.
Ligações à portas: As aplicações devem ligar-se nativamente a uma porta e ouvir conexões. Roteamento e encaminhamento de solicitações devem ser tratados externamente.
Concorrência: As aplicações devem confiar na escalabilidade através do modelo de processo. A execução de várias cópias de uma aplicação simultaneamente, potencialmente em vários servidores, permite o escalonamento sem ajustar o código da aplicação.
Descartabilidade: Os processos devem ser capazes de iniciar rapidamente e parar normalmente sem efeitos colaterais sérios.
Paridade Desenv/prod: Seus ambientes de teste, preparação e produção devem ter uma correspondência próxima e ser mantidos em sincronia. Diferenças entre ambientes são oportunidades para incompatibilidades e configurações não testadas aparecerem.
Logs: As aplicações devem transmitir os logs para a saída padrão, para que os serviços externos possam decidir a melhor forma de lidar com eles.
Processos administrativos: Processos de administração únicos devem ser executados em releases específicos e enviados com o código do processo principal.

Ao aderir às diretrizes fornecidas pelos Doze Fatores, você pode criar e executar aplicativos usando um modelo adequado ao ambiente de execução do Kubernetes. Os Doze Fatores encorajam os desenvolvedores a se concentrarem na responsabilidade principal de suas aplicações, a considerarem as condições operacionais e as interfaces entre os componentes e a usarem entradas, saídas e recursos de gerenciamento de processos padrão, para serem executados de maneira previsível no Kubernetes.

Containerizando Componentes da Aplicação

O Kubernetes usa containers para executar aplicações empacotadas isoladas em seus nodes de cluster. Para executar no Kubernetes, suas aplicações devem ser encapsuladas em uma ou mais imagens de container e executadas usando um runtime de container como o Docker. Embora containerizar seus componentes seja um requisito para o Kubernetes, isso também ajuda a reforçar muitos dos princípios da metodologia de doze fatores para aplicações, discutidos acima, permitindo fácil escalonamento e gerenciamento.

Por exemplo, os containers fornecem isolamento entre o ambiente da aplicação e o sistema do host externo, suporta uma abordagem em rede, orientada a serviços para comunicação entre aplicações, e normalmente busca a configuração através de variáveis de ambiente e expõe logs gravados na saída de erro e na saída padrão. Os próprios containers encorajam a concorrência baseada em processos e ajudam a manter a paridade desenv/prod sendo escaláveis independentemente e agrupando o ambiente de runtime do processo. Essas características tornam possível empacotar suas aplicações para que elas funcionem sem problemas no Kubernetes.

Diretrizes para Otimizar Containers

A flexibilidade da tecnologia de container permite muitas maneiras diferentes de encapsular uma aplicação. Contudo, alguns métodos funcionam melhor do que outros em um ambiente Kubernetes.

A maioria das boas práticas recomendadas na containerização de suas aplicações tem a ver com a construção de imagens, onde você define como seu software será configurado e executado dentro de um container. Em geral, manter tamanhos de imagem pequenos e compostos oferece vários benefícios. Imagens de tamanho otimizado podem reduzir o tempo e os recursos necessários para iniciar um novo container em um cluster, mantendo o perfil gerenciável e reutilizando as camadas existentes entre as atualizações de imagem.

Um bom primeiro passo ao criar imagens de container é fazer o seu melhor para separar suas etapas de criação da imagem final que será executada na produção. Construir software geralmente requer ferramental extra, toma tempo adicional e produz artefatos que podem ser inconsistentes de container para container ou desnecessários para o runtime de execução final, dependendo do ambiente. Uma maneira de separar claramente o processo de construção do runtime de execução é usar as construções de vários estágios do Docker. As configurações das construções de vários estágios permitem que você especifique uma imagem de base para usar durante o processo de construção e defina outra para usar no runtime. Isso possibilita a criação de software usando uma imagem com todas as ferramentas de compilação instaladas e copiar os artefatos resultantes para uma imagem simplificada que será usada a cada vez, posteriormente.

Com esse tipo de funcionalidade disponível, geralmente é uma boa ideia criar imagens de produção em cima de uma imagem matriz mínima. Se você quiser evitar completamente o inchaço encontrado em camadas da imagem matriz do estilo "distro" como ubuntu:16.04 (que inclui um ambiente Ubuntu 16.04 bastante completo), você pode construir suas imagens com o scratch — A imagem de base mais minimalista do Docker — como matriz. Contudo, a camada base scratch não fornece acesso a muitas das ferramentas principais e muitas vezes quebrará as suposições sobre o ambiente que alguns softwares detêm. Como uma alternativa, a imagem alpine Alpine Linux ganhou poularidade por ser um ambiente de base sólido e mínimo, que fornece uma distribuição Linux minúscula, mas com recursos completos.

Para linguagens interpretadas como o Python ou Ruby, o paradigma muda levemente, já que não há um estágio de compilação e o interpretador deve estar disponível para executar o código em produção. No entanto, como as imagens magras ainda são ideais, muitas imagens otimizadas para linguagens específicas, construídas a partir do Alpine Linux estão disponíveis no Docker Hub. Os benefícios de se usar uma imagem menor para linguagens interpretadas são semelhantes aos das linguagens compiladas: O Kubernetes poderá buscar rapidamente todas as imagens de container necessárias para novos nodes para começar a fazer trabalho significativo.

Decidindo sobre o Escopo para Containers e Pods

Embora suas aplicações devam ser containerizadas para executar em um cluster Kubernetes, os pods são a menor unidade de abstração que o Kubernetes pode gerenciar diretamente. Um pod é um objeto Kubernetes composto de um ou mais containers fortemente acoplados. os containers em um pod compartilham um ciclo de vida e são gerenciados juntos como uma única unidade. Por exemplo, o scheduling desses containers é feito sempre no mesmo node, são iniciados e parados em sincronia e compartilham recursos como sistemas de arquivos e espaço de IP.

No início, pode ser difícil descobrir a melhor maneira de dividir suas aplicações em containers e pods. Isso torna importante entender como o Kubernetes lida com esses componentes e o que cada camada de abstração fornece para seus sistemas. Algumas considerações podem ajudá-lo a identificar alguns pontos naturais de encapsulamento para sua aplicação com cada uma dessas abstrações.

Uma maneira de determinar um escopo efetivo para seus containers é procurar por limites naturais do desenvolvimento. Se seus sistemas operam utilizando a arquitetura de microservices, containers bem projetados são frequentemente construídos para representar unidades discretas de funcionalidade que podem ser usadas em uma variedade de contextos. Esse nível de abstração permite à sua equipe lançar alterações nas imagens de containers e, em seguida, fazer o deploy dessa nova funcionalidade em qualquer ambiente onde essas imagens sejam utilizadas. As aplicações podem ser construídas através da composição de containers individuais que preencham uma determinada função, mas podem não realizar um processo inteiro sozinhos.

Em contraste com o exposto cima, os pods são geralmente construídos pensando em quais partes do seu sistema podem se beneficiar mais do gerenciamento independente. Como o Kubernetes usa pods como sua menor abstração de interação com o usuário, essas são as unidades mais primitivas com as quais as ferramentas e a API do Kubernetes podem interagir e controlar diretamente. Você pode iniciar, parar e reiniciar pods, ou utilizar objetos de nível superior construídos em pods para introduzir recursos de replicação e gerenciamento do ciclo e vida. O Kubernetes não lhe permite gerenciar os containers dentro de um pod de forma independente, portanto, você não deve agrupar containers que possam se beneficiar de uma administração separada.

Como muitos dos recursos e abstrações do Kubernetes lidam diretamente com os pods, faz sentido agrupar itens que devem ser escalados juntos em um único pod e separar aqueles que devem ser escalados independentemente. Por exemplo, separar seus servidores web de seus servidores de aplicação em diferentes pods permite escalar cada camada de forma independente, conforme necessário. No entanto, o empacotamento de um servidor web e de um adaptador de banco de dados no mesmo conjunto pode fazer sentido se o adaptador fornecer a funcionalidade essencial que o servidor web precisa para funcionar corretamente.

Melhorando a Funcionalidade do Pod através do Empacotamento de Containers de Apoio

Com isso em mente, quais tipos de containers devem ser agrupados em um único pod? Geralmente, um container primário é responsável pelo cumprimento das funções principais do pod, mas podem ser definidos containers adicionais que modifiquem ou estendam o container principal ou o ajudem a se conectar a um ambiente de deployment exclusivo.

Por exemplo, em um pod de servidor web, um container Nginx pode escutar solicitações e fornecer conteúdo enquanto um container associado atualiza arquivos estáticos quando um repositório é alterado. Pode ser tentador empacotar esses dois componentes em um único container, mas há benefícios significativos em implementá-los como containers separados. O container do servidor web e o atualizador do repositório podem ser usados independentemente em diferentes contextos. Eles podem ser mantidos por equipes diferentes e cada um deles podem ser desenvolvidos para generalizar seu comportamento para trabalhar com diferentes containers complementares.

Brendan Burns e David Oppenheimer identificaram três padrões primários para empacotar containers de apoio em seus artigos em design patterns for container-based distributed systems. Estes representam alguns dos casos de uso mais comuns para empacotamento de containers juntos em um pod:

Sidecar: Neste padrão, o container secundário estende e aprimora a funcionalidade principal do container primário. Esse padrão envolve a execução de funções não padronizadas ou utilitárias em um container separado. Por exemplo, um container que encaminha logs ou monitora atualizações em valores de configuração pode aumentar a funcionalidade de um pod sem alterar significativamente seu foco principal.
Ambassador: O padrão ambassador usa um container suplementar para abstrair recursos remotos para o container principal. O container principal se conecta diretamente ao container ambassador que, por sua vez, se conecta e abstrai os pools de recursos externos potencialmente complexos, como um cluster de Redis distribuído. O container principal não precisa saber ou se preocupar com o ambiente de deployment real para se conectar a serviços externos.
Adaptor: O padrão adaptor é usado para converter os dados, protocolos ou interfaces do container primário para alinhar com os padrões esperados por terceiros. Containers Adaptor permitem o acesso uniforme a serviços centralizados, mesmo quando as aplicações que eles atendem podem suportar nativamente somente interfaces incompatíveis.

Extraindo a Configuração em ConfigMaps e Secrets

Embora a configuração da aplicação possa ser posta em imagens de container, é melhor tornar seus componentes configuráveis no runtime para suportar deployments em vários contextos e permitir uma administração mais flexível. Para gerenciar parâmetros de configuração de runtime, o Kubernetes usa dois objetos chamados ConfigMaps e Secrets.

ConfigMaps é um mecanismo usado para armazenar dados que podem ser expostos a pods e outros objetos em runtime. Os dados armazenados dentro de ConfigMaps podem ser apresentados como variáveis de ambiente ou montados como arquivos no pod. Ao projetar suas aplicações para ler esses locais, você pode injetar a configuração no runtime usando o ConfigMaps e modificar o comportamento de seus componentes sem precisar reconstruir a imagem do container.

Secrets são um tipo similar de objeto Kubernetes usado para armazenar dados sigilosos com segurança e seletivamente permitir o acesso a pods e outros componentes conforme necessário. Os secrets são uma maneira conveniente de transmitir material confidencial para aplicações sem armazená-los como texto simples em locais facilmente acessíveis em sua configuração normal. Funcionalmente, eles trabalham da mesma maneira que os ConfigMaps, portanto, as aplicações podem consumir dados de ConfigMaps e Secrets usando os mesmos mecanismos.

ConfigMaps e Secrets o ajudam a evitar colocar a configuração diretamente nas definições de objeto do Kubernetes. Você pode mapear a chave de configuração em vez do valor, permitindo que você atualize a configuração dinamicamente, através da modificação do ConfigMap ou do Secret. Isso lhe dá a oportunidade de alterar o comportamento do runtime ativo dos pods e outros objetos do Kubernetes sem modificar as definições dos recursos do Kubernetes.

Implementando as provas Readiness e Liveness

O Kubernetes inclui uma grande quantidade de funcionalidades prontas para uso para gerenciar ciclos de vida e garantir que suas aplicações estejam sempre íntegras e disponíveis. No entanto, para aproveitar esses recursos, o Kubernetes precisa entender como ele deve monitorar e interpretar a integridade da sua aplicação. Para fazer isto, o Kubernetes lhe permite definir provas liveness e readiness.

Provas Liveness permitem que o Kubernetes determine se uma aplicação em um container está no ar e funcionando ativamente. O Kubernetes pode executar comandos periodicamente dentro do container para verificar o comportamento básico da aplicação ou pode enviar solicitações de rede HTTP ou TCP para um local designado para determinar se o processo está disponível e é capaz de responder conforme o esperado. Se uma prova liveness falha, o Kubernetes reinicia o container para tentar restabelecer a funcionalidade dentro do pod.

Provas Readness são um ferramenta similar usada para determinar se um pod está pronto para servir o tráfego. As aplicações em um container podem precisar executar procedimentos de inicialização antes de estarem prontas para aceitar solicitações de clientes ou podem precisar ser recarregadas ao serem notificadas sobre uma nova configuração. Quando uma prova readness falha, em vez de reiniciar o container, O Kubernetes pára de enviar temporariamente solicitações ao pod. Isso permite que o pod complete suas rotinas de inicialização ou manutenção sem afetar a integridade do grupo como um todo.

Ao combinar as provas liveness e readiness, você pode instruir o Kubernetes a reiniciar automaticamente os pods ou removê-los dos grupos de back-end. Configurar sua infraestrutura para aproveitar esses recursos permite que o Kubernetes gerencie a disponibilidade e a integridade de suas aplicações sem trabalho operacional adicional.

Usando Deployments para Gerenciar Escalabilidade e Disponibilidade

Mais cedo, ao discutir alguns fundamentos de projetos de pods, também mencionamos que outros objetos do Kubernetes são construídos com base nessas primitivas para fornecer funcionalidade mais avançada. Um deployment, um desses objetos compostos, é provavelmente o objeto Kubernetes mais comumente definido e manipulado.

Deployments são objetos compostos que se baseiam em outras primativas do Kubernetes para somar recursos adicionais. Eles adicionam recursos de gerenciamento de ciclo de vida a objetos intermediários chamados replicasets, como a capacidade de executar atualizações contínuas, rollback para versões anteriores e transição entre estados. Esses replicasets permitem que você defina modelos de pod para lançar e gerenciar várias cópias de um único projeto de pod. Isso ajuda você a escalar facilmente sua infraestrutura, gerenciar os requisitos de disponibilidade e reiniciar automaticamente os pods em caso de falha.

Esses recursos adicionais fornecem uma estrutura administrativa e recursos de autocorreção para a abstração de pods relativamente simples. Embora os pods sejam as unidades que executam as cargas de trabalho que você define, eles não são as unidades que você geralmente deve provisionar e gerenciar. Em vez disso, pense nos pods como um bloco construtivo que pode executar aplicativos de maneira robusta quando provisionados por meio de objetos de nível mais alto, como os deployments.

Criando Regras de Serviços e de Ingresso para Gerenciar o Acesso às Camadas de Aplicação

Deployments lhe permitem provisionar e gerenciar conjuntos de pods intercambiáveis para escalar suas aplicações e atender às demandas do usuário. No entanto, o roteamento de tráfego para os pods provisionados é uma preocupação à parte. Como os pods são trocados como parte de atualizações contínuas, reiniciados ou movidos devido a falhas de host, os endereços de rede associados anteriormente ao grupo em execução serão alterados. Os serviços do Kubernetes lhe permitem gerenciar essa complexidade mantendo as informações de roteamento para pools dinâmicos de pods e controlando o acesso a várias camadas da sua infraestrutura.

No Kubernetes, serviços são mecanismos específicos que controlam como o tráfego é roteado para conjuntos de pods. Seja encaminhando tráfego de clientes externos ou gerenciando conexões entre vários componentes internos, os serviços permitem que você controle como o tráfego deve fluir. O Kubernetes atualizará e manterá todas as informações necessárias para encaminhar conexões para os pods relevantes, mesmo que o ambiente se altere e o cenário de rede mude.

Acessando Serviços Internamente

Para usar os serviços efetivamente, primeiro você deve determinar os consumidores presumidos para cada grupo de pods. Se o seu serviço só será usado por outras aplicações implantadas em seu cluster do Kubernetes, o tipo de serviço clusterIP permite que você se conecte a um conjunto de pods usando um endereço IP estável que só pode ser roteado de dentro do cluster. Qualquer objeto com deploy no cluster pode se comunicar com o grupo de pods replicados enviando tráfego diretamente para o endereço IP do serviço. Esse é o tipo de serviço mais simples, que funciona bem para camadas internas das aplicações.

Um add-on opcional de DNS permite que o Kubernetes forneça nomes DNS para serviços. Isso permite que os pods e outros objetos se comuniquem com os serviços pelo nome, em vez do endereço IP. Esse mecanismo não altera significativamente o uso do serviço, mas os identificadores baseados em nome podem simplificar a conexão de componentes ou a definição de interações sem conhecer o endereço IP do serviço antecipadamente.

Expondo Serviços para Consumo Público

Se a interface deve ser publicamente acessível, sua melhor opção geralmente é o tipo de serviço load balancer. Ele usa a API do seu provedor de nuvem específico para provisionar um balanceador de carga, que serve tráfego para os pods de serviço por meio de um endereço IP exposto publicamente. Isso lhe permite rotear solicitações externas para os pods em seu serviço, oferecendo um canal de rede controlado para sua rede interna de clusters.

Como o tipo de serviço load balancer cria um balanceador de carga para cada serviço, pode se tornar caro expor publicamente os serviços do Kubernetes usando esse método. Para ajudar a aliviar isso, os objetos ingress do Kubernetes podem ser usados para descrever como rotear diferentes tipos de solicitações para diferentes serviços com base em um conjunto predeterminado de regras. Por exemplo, solicitações para "example.com" poderiam ir para o serviço A, enquanto solicitações para "sammytheshark.com" poderiam ser roteadas para o serviço B. Os objetos ingress fornecem uma maneira de descrever como rotear logicamente um fluxo misto de solicitações para seus serviços de destino com base em padrões predefinidos.

Regras Ingress devem ser interpretadas por um ingress controller — normalmente algum tipo de balanceamento de carga, como o Nginx — implantado no cluster como um pod, que implementa as regras de entrada e encaminha o tráfego apropriadamente para os serviços do Kubernetes. Atualmente, o tipo de objeto ingress está na versão beta, mas há várias implementações de trabalho que podem ser usadas para minimizar o número de balanceadores de carga externos que os proprietários de cluster precisam executar.

Usando Sintaxe Declarativa para Gerenciar o Estado do Kubernetes

O Kubernetes oferece bastante flexibilidade na definição e controle dos recursos implantados (com deploy realizado) em seu cluster. Utilizando ferramentas como o kubectl, você pode definir imperativamente objetos ad-hoc para fazer deploy imediatamente em seu cluster. Embora isso possa ser útil para fazer o deploy de recursos rapidamente ao aprender o Kubernetes, há desvantagens nessa abordagem que a tornam indesejável para a administração de produção a longo prazo.

Um dos principais problemas com o gerenciamento imperativo é que ele não deixa nenhum registro das alterações que você fez deploy em seu cluster. Isso dificulta ou impossibilita a recuperação no caso de falhas ou para acompanhar as mudanças operacionais à medida que são aplicadas aos seus sistemas.

Felizmente, o Kubernetes fornece uma sintaxe declarativa alternativa que lhe permite definir recursos em arquivos de texto e em seguida usar o kubectl para aplicar a configuração ou alteração. Armazenar esses arquivos de configuração em um repositório de controle de versão é uma maneira simples de monitorar alterações e integrar com os processos de revisão usados para outras áreas de sua organização. O gerenciamento baseado em arquivos também simplifica a adaptação de padrões existentes para novos recursos, copiando e editando definições existentes. Armazenar suas definições de objeto do Kubernetes em diretórios versionados permite manter um instantâneo do estado do cluster desejado em cada instante no tempo. Isso pode ter um valor inestimável durante operações de recuperação, migrações ou ao rastrear a causa raiz de alterações não intencionais introduzidas em seu sistema.

Conclusão

Gerenciar a infraestrutura que executará suas aplicações e aprender como aproveitar melhor os recursos oferecidos pelos ambientes modernos de orquestração pode ser assustador. No entanto, muitos dos benefícios oferecidos por sistemas como o Kubernetes e tecnologias como containers ficam mais claros quando suas práticas de desenvolvimento e operações se alinham aos conceitos sobre os quais o ferramental é construído. Arquitetar seus sistemas usando os padrões nos quais o Kubernetes se destaca e entender como certos recursos podem aliviar alguns dos desafios associados a deployments altamente complexos pode ajudar a melhorar sua experiência com a execução na plataforma.

How To Install Apache Kafka on CentOS 7

2018-08-03T15:02:10Z

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

Apache Kafka is a popular distributed message broker designed to efficiently handle large volumes of real-time data. A Kafka cluster is not only highly scalable and fault-tolerant, but it also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a publish/subscribe messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.

A publish/subscribe messaging system allows one or more producers to publish messages without considering the number of consumers or how they will process the messages. Subscribed clients are notified automatically about updates and the creation of new messages. This system is more efficient and scalable than systems where clients poll periodically to determine if new messages are available.

In this tutorial, you will install and use Apache Kafka 1.1.0 on CentOS 7.

Prerequisites

To follow along, you will need:

One CentOS 7 server and a non-root user with sudo privileges. Follow the steps specified in this guide if you do not have a non-root user set up.
At least 4GB of RAM on the server. Installations without this amount of RAM may cause the Kafka service to fail, with the Java virtual machine (JVM) throwing an "Out Of Memory" exception during startup.
OpenJDK 8 installed on your server. To install this version, follow these instructions on installing specific versions of OpenJDK. Kafka is written in Java, so it requires a JVM; however, its startup shell script has a version detection bug that causes it to fail to start with JVM versions above 8.

Step 1 — Creating a User for Kafka

Since Kafka can handle requests over a network, you should create a dedicated user for it. This minimizes damage to your CentOS machine should the Kafka server be compromised. We will create a dedicated kafka user in this step, but you should create a different non-root user to perform other tasks on this server once you have finished setting up Kafka.

Logged in as your non-root sudo user, create a user called kafka with the useradd command:

sudo useradd kafka -m

The -m flag ensures that a home directory will be created for the user. This home directory, /home/kafka, will act as our workspace directory for executing commands in the sections below.

Set the password using passwd:

sudo passwd kafka

Add the kafka user to the wheel group with the adduser command, so that it has the privileges required to install Kafka's dependencies:

sudo usermod -aG wheel kafka

Your kafka user is now ready. Log into this account using su:

su -l kafka

Now that we've created the Kafka-specific user, we can move on to downloading and extracting the Kafka binaries.

Step 2 — Downloading and Extracting the Kafka Binaries

Let's download and extract the Kafka binaries into dedicated folders in our kafka user's home directory.

To start, create a directory in /home/kafka called Downloads to store your downloads:

mkdir ~/Downloads

Use curl to download the Kafka binaries:

curl "http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.12-1.1.0.tgz" -o ~/Downloads/kafka.tgz

Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation:

mkdir ~/kafka && cd ~/kafka

Extract the archive you downloaded using the tar command:

tar -xvzf ~/Downloads/kafka.tgz --strip 1

We specify the --strip 1 flag to ensure that the archive's contents are extracted in ~/kafka/ itself and not in another directory (such as ~/kafka/kafka_2.12-1.1.0/) inside of it.

Now that we've downloaded and extracted the binaries successfully, we can move on configuring to Kafka to allow for topic deletion.

Step 3 — Configuring the Kafka Server

Kafka's default behavior will not allow us to delete a topic, the category, group, or feed name to which messages can be published. To modify this, let's edit the configuration file.

Kafka's configuration options are specified in server.properties. Open this file with vi or your favorite editor:

vi ~/kafka/config/server.properties

Let's add a setting that will allow us to delete Kafka topics. Press i to insert text, and add the following to the bottom of the file:

~/kafka/config/server.properties

delete.topic.enable = true

When you are finished, press ESC to exit insert mode and :wq to write the changes to the file and quit. Now that we've configured Kafka, we can move on to creating systemd unit files for running and enabling it on startup.

Step 4 — Creating Systemd Unit Files and Starting the Kafka Server

In this section, we will create systemd unit files for the Kafka service. This will help us perform common service actions such as starting, stopping, and restarting Kafka in a manner consistent with other Linux services.

Zookeeper is a service that Kafka uses to manage its cluster state and configurations. It is commonly used in many distributed systems as an integral component. If you would like to know more about it, visit the official Zookeeper docs.

Create the unit file for zookeeper:

sudo vi /etc/systemd/system/zookeeper.service

Enter the following unit definition into the file:

/etc/systemd/system/zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that Zookeeper requires networking and the filesystem to be ready before it can start.

The [Service] section specifies that systemd should use the zookeeper-server-start.sh and zookeeper-server-stop.sh shell files for starting and stopping the service. It also specifies that Zookeeper should be restarted automatically if it exits abnormally.

Next, create the systemd service file for kafka:

sudo vi /etc/systemd/system/kafka.service

Enter the following unit definition into the file:

/etc/systemd/system/kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that this unit file depends on zookeeper.service. This will ensure that zookeeper gets started automatically when the kafa service starts.

The [Service] section specifies that systemd should use the kafka-server-start.sh and kafka-server-stop.sh shell files for starting and stopping the service. It also specifies that Kafka should be restarted automatically if it exits abnormally.

Now that the units have been defined, start Kafka with the following command:

sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:

journalctl -u kafka

You should see output similar to the following:

OutputJul 17 18:38:59 kafka-centos systemd[1]: Started kafka.service.

You now have a Kafka server listening on port 9092.

While we have started the kafka service, if we were to reboot our server, it would not be started automatically. To enable kafka on server boot, run:

sudo systemctl enable kafka

Now that we've started and enabled the services, let's check the installation.

Step 5 — Testing the Installation

Let's publish and consume a "Hello World" message to make sure the Kafka server is behaving correctly. Publishing messages in Kafka requires:

A producer, which enables the publication of records and data to topics.
A consumer, which reads messages and data from topics.

First, create a topic named TutorialTopic by typing:

~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

You can create a producer from the command line using the kafka-console-producer.sh script. It expects the Kafka server's hostname, port, and a topic name as arguments.

Publish the string "Hello, World" to the TutorialTopic topic by typing:

echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null

Next, you can create a Kafka consumer using the kafka-console-consumer.sh script. It expects the ZooKeeper server's hostname and port, along with a topic name as arguments.

The following command consumes messages from TutorialTopic. Note the use of the --from-beginning flag, which allows the consumption of messages that were published before the consumer was started:

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning

If there are no configuration issues, you should see Hello, World in your terminal:

OutputHello, World

The script will continue to run, waiting for more messages to be published to the topic. Feel free to open a new terminal and start a producer to publish a few more messages. You should be able to see them all in the consumer's output.

When you are done testing, press CTRL+C to stop the consumer script. Now that we have tested the installation, let's move on to installing KafkaT.

Step 6 — Installing KafkaT (Optional)

KafkaT is a tool from Airbnb that makes it easier for you to view details about your Kafka cluster and perform certain administrative tasks from the command line. Because it is a Ruby gem, you will need Ruby to use it. You will also need ruby-devel and build-related packages such as make and gcc to be able to build the other gems it depends on. Install them using yum:

sudo yum install ruby ruby-devel make gcc patch

You can now install KafkaT using the gem command:

sudo gem install kafkat

KafkaT uses .kafkatcfg as the configuration file to determine the installation and log directories of your Kafka server. It should also have an entry pointing KafkaT to your ZooKeeper instance.

Create a new file called .kafkatcfg:

vi ~/.kafkatcfg

Add the following lines to specify the required information about your Kafka server and Zookeeper instance:

~/.kafkatcfg

{
  "kafka_path": "~/kafka",
  "log_path": "/tmp/kafka-logs",
  "zk_path": "localhost:2181"
}

You are now ready to use KafkaT. For a start, here's how you would use it to view details about all Kafka partitions:

kafkat partitions

You will see the following output:

OutputTopic                 Partition   Leader      Replicas        ISRs    
TutorialTopic         0             0         [0]             [0]
__consumer_offsets    0             0         [0]                           [0]
...
...

You will see TutorialTopic, as well as __consumer_offsets, an internal topic used by Kafka for storing client-related information. You can safely ignore lines starting with __consumer_offsets.

To learn more about KafkaT, refer to its GitHub repository.

Step 7 — Setting Up a Multi-Node Cluster (Optional)

If you want to create a multi-broker cluster using more CentOS 7 machines, you should repeat Step 1, Step 4, and Step 5 on each of the new machines. Additionally, you should make the following changes in the server.properties file for each:

The value of the broker.id property should be changed such that it is unique throughout the cluster. This property uniquely identifies each server in the cluster and can have any string as its value. For example, "server1", "server2", etc.
The value of the zookeeper.connect property should be changed such that all nodes point to the same ZooKeeper instance. This property specifies the Zookeeper instance's address and follows the <HOSTNAME/IP_ADDRESS>:<PORT> format. For example, "203.0.113.0:2181", "203.0.113.1:2181" etc.

If you want to have multiple ZooKeeper instances for your cluster, the value of the zookeeper.connect property on each node should be an identical, comma-separated string listing the IP addresses and port numbers of all the ZooKeeper instances.

Step 8 — Restricting the Kafka User

Now that all of the installations are done, you can remove the kafka user's admin privileges. Before you do so, log out and log back in as any other non-root sudo user. If you are still running the same shell session you started this tutorial with, simply type exit.

Remove the kafka user from the sudo group:

sudo gpasswd -d kafka wheel

To further improve your Kafka server's security, lock the kafka user's password using the passwd command. This makes sure that nobody can directly log into the server using this account:

sudo passwd kafka -l

At this point, only root or a sudo user can log in as kafka by typing in the following command:

sudo su - kafka

In the future, if you want to unlock it, use passwd with the -u option:

sudo passwd kafka -u

You have now successfully restricted the kafka user's admin privileges.

Conclusion

You now have Apache Kafka running securely on your CentOS server. You can make use of it in your projects by creating Kafka producers and consumers using Kafka clients, which are available for most programming languages. To learn more about Kafka, you can also consult its documentation.

An Introduction to Helm, the Package Manager for Kubernetes

2018-08-02T16:08:59Z

Introduction

Deploying applications to Kubernetes – the powerful and popular container-orchestration system – can be complex. Setting up a single application can involve creating multiple interdependent Kubernetes resources – such as pods, services, deployments, and replicasets – each requiring you to write a detailed YAML manifest file.

Helm is a package manager for Kubernetes that allows developers and operators to more easily package, configure, and deploy applications and services onto Kubernetes clusters.

Helm is now an official Kubernetes project and is part of the Cloud Native Computing Foundation, a non-profit that supports open source projects in and around the Kubernetes ecosystem.

In this article we will give an overview of Helm and the various abstractions it uses to simplify deploying applications to Kubernetes. If you are new to Kubernetes, it may be helpful to read An Introduction to Kubernetes first to familiarize yourself with the basics concepts.

An Overview of Helm

Most every programming language and operating system has its own package manager to help with the installation and maintenance of software. Helm provides the same basic feature set as many of the package managers you may already be familiar with, such as Debian's apt, or Python's pip.

Helm can:

Install software.
Automatically install software dependencies.
Upgrade software.
Configure software deployments.
Fetch software packages from repositories.

Helm provides this functionality through the following components:

A command line tool, helm, which provides the user interface to all Helm functionality.
A companion server component, tiller, that runs on your Kubernetes cluster, listens for commands from helm, and handles the configuration and deployment of software releases on the cluster.
The Helm packaging format, called charts.
An official curated charts repository with prepackaged charts for popular open-source software projects.

We'll investigate the charts format in more detail next.

Charts

Helm packages are called charts, and they consist of a few YAML configuration files and some templates that are rendered into Kubernetes manifest files. Here is the basic directory structure of a chart:

Example chart directory

package-name/
  charts/
  templates/
  Chart.yaml
  LICENSE
  README.md
  requirements.yaml
  values.yaml

These directories and files have the following functions:

charts/: Manually managed chart dependencies can be placed in this directory, though it is typically better to use requirements.yaml to dynamically link dependencies.
templates/: This directory contains template files that are combined with configuration values (from values.yaml and the command line) and rendered into Kubernetes manifests. The templates use the Go programming language's template format.
Chart.yaml: A YAML file with metadata about the chart, such as chart name and version, maintainer information, a relevant website, and search keywords.
LICENSE: A plaintext license for the chart.
README.md: A readme file with information for users of the chart.
requirements.yaml: A YAML file that lists the chart's dependencies.
values.yaml: A YAML file of default configuration values for the chart.

The helm command can install a chart from a local directory, or from a .tar.gz packaged version of this directory structure. These packaged charts can also be automatically downloaded and installed from chart repositories or repos.

We'll look at chart repositories next.

Chart Repositories

A Helm chart repo is a simple HTTP site that serves an index.yaml file and .tar.gz packaged charts. The helm command has subcommands available to help package charts and create the required index.yaml file. These files can be served by any web server, object storage service, or a static site host such as GitHub Pages.

Helm comes preconfigured with a default chart repository, referred to as stable. This repo points to a Google Storage bucket at https://kubernetes-charts.storage.googleapis.com. The source for the stable repo can be found in the helm/charts Git repository on GitHub.

Alternate repos can be added with the helm repo add command. Some popular alternate repositories are:

The official incubator repo that contains charts that are not yet ready for stable. Instructions for using incubator can be found on the official Helm charts GitHub page.
Bitnami Helm Charts which provide some charts that aren't covered in the official stable repo.

Whether you're installing a chart you've developed locally, or one from a repo, you'll need to configure it for your particular setup. We'll look into configs next.

Chart Configuration

A chart usually comes with default configuration values in its values.yaml file. Some applications may be fully deployable with default values, but you'll typically need to override some of the configuration to meet your needs.

The values that are exposed for configuration are determined by the author of the chart. Some are used to configure Kubernetes primitives, and some may be passed through to the underlying container to configure the application itself.

Here is a snippet of some example values:

values.yaml

service:
  type: ClusterIP
  port: 3306

These are options to configure a Kubernetes Service resource. You can use helm inspect values chart-name to dump all of the available configuration values for a chart.

These values can be overridden by writing your own YAML file and using it when running helm install, or by setting options individually on the command line with the --set flag. You only need to specify those values that you want to change from the defaults.

A Helm chart deployed with a particular configuration is called a release. We will talk about releases next.

Releases

During the installation of a chart, Helm combines the chart's templates with the configuration specified by the user and the defaults in value.yaml. These are rendered into Kubernetes manifests that are then deployed via the Kubernetes API. This creates a release, a specific configuration and deployment of a particular chart.

This concept of releases is important, because you may want to deploy the same application more than once on a cluster. For instance, you may need multiple MySQL servers with different configurations.

You also will probably want to upgrade different instances of a chart individually. Perhaps one application is ready for an updated MySQL server but another is not. With Helm, you upgrade each release individually.

You might upgrade a release because its chart has been updated, or because you want to update the release's configuration. Either way, each upgrade will create a new revision of a release, and Helm will allow you to easily roll back to previous revisions in case there's an issue.

Creating Charts

If you can't find an existing chart for the software you are deploying, you may want to create your own. Helm can output the scaffold of a chart directory with helm create chart-name. This will create a folder with the files and directories we discussed in the Charts section above.

From there, you'll want to fill out your chart's metadata in Chart.yaml and put your Kubernetes manifest files into the templates directory. You'll then need to extract relevant configuration variables out of your manifests and into values.yaml, then include them back into your manifest templates using the templating system.

The helm command has many subcommands available to help you test, package, and serve your charts. For more information, please read the official Helm documentation on developing charts.

Conclusion

In this article we reviewed Helm, the package manager for Kubernetes. We overviewed the Helm architecture and the individual helm and tiller components, detailed the Helm charts format, and looked at chart repositories. We also looked into how to configure a Helm chart and how configurations and charts are combined and deployed as releases on Kubernetes clusters. Finally, we touched on the basics of creating a chart when a suitable chart isn't already available.

For more information about Helm, take a look at the official Helm documentation. To find official charts for Helm, check out the official helm/charts Git repository on GitHub.

How To Install the Django Web Framework on Ubuntu 18.04

2018-08-02T19:53:11Z

Introduction

Django is a full-featured Python web framework for developing dynamic websites and applications. Using Django, you can quickly create Python web applications and rely on the framework to do a good deal of the heavy lifting.

In this guide, you will get Django up and running on an Ubuntu 18.04 server. After installation, you will start a new project to use as the basis for your site.

Different Methods

There are different ways to install Django, depending upon your needs and how you want to configure your development environment. These have different advantages and one method may lend itself better to your specific situation than others.

Some of the different methods include:

Global install from packages: The official Ubuntu repositories contain Django packages that can be installed with the conventional apt package manager. This is simple, but not as flexible as some other methods. Also, the version contained in the repositories may lag behind the official versions available from the project.
Install with pip in a virtual environment: You can create a self-contained environment for your projects using tools like venv and virtualenv. A virtual environment allows you to install Django in a project directory without affecting the larger system, along with other per-project customizations and packages. This is typically the most practical and recommended approach to working with Django.
Development version install with git: If you wish to install the latest development version instead of the stable release, you can acquire the code from the Git repo. This is necessary to get the latest features/fixes and can be done within your virtual environment. Development versions do not have the same stability guarantees as more stable versions, however.

Prerequisites

Before you begin, you should have a non-root user with sudo privileges available on your Ubuntu 18.04 server. To set this up, follow our Ubuntu 18.04 initial server setup guide.

Global Install from Packages

If you wish to install Django using the Ubuntu repositories, the process is very straightforward.

First, update your local package index with apt:

sudo apt update

Next, check which version of Python you have installed. 18.04 ships with Python 3.6 by default, which you can verify by typing:

python3 -V

You should see output like this:

Output
Python 3.6.5

Next, install Django:

sudo apt install python3-django

You can test that the installation was successful by typing:

django-admin --version

Output
1.11.11

This means that the software was successfully installed. You may also notice that the Django version is not the latest stable version. To learn more about how to use the software, skip ahead to learn how to create sample project.

Install with pip in a Virtual Environment

The most flexible way to install Django on your system is within a virtual environment. We will show you how to install Django in a virtual environment that we will create with the venv module, part of the standard Python 3 library. This tool allows you to create virtual Python environments and install Python packages without affecting the rest of the system. You can therefore select Python packages on a per-project basis, regardless of conflicts with other projects' requirements.

Let's begin by refreshing the local package index:

sudo apt update

Check the version of Python you have installed:

python3 -V

Output
Python 3.6.5

Next, let's install pip from the Ubuntu repositories:

sudo apt install python3-pip

Once pip is installed, you can use it to install the venv package:

sudo apt install python3-venv

Now, whenever you start a new project, you can create a virtual environment for it. Start by creating and moving into a new project directory:

mkdir ~/newproject
cd ~/newproject

Next, create a virtual environment within the project directory using the python command that's compatible with your version of Python. We will call our virtual environment my_env, but you should name it something descriptive:

python3.6 -m venv my_env

This will install standalone versions of Python and pip into an isolated directory structure within your project directory. A directory will be created with the name you select, which will hold the file hierarchy where your packages will be installed.

To install packages into the isolated environment, you must activate it by typing:

source my_env/bin/activate

Your prompt should change to reflect that you are now in your virtual environment. It will look something like (my_env)username@hostname:~/newproject$.

In your new environment, you can use pip to install Django. Regardless of your Python version, pip should just be called pip when you are in your virtual environment. Also note that you do not need to use sudo since you are installing locally:

pip install django

You can verify the installation by typing:

django-admin --version

Output
2.1

Note that your version may differ from the version shown here.

To leave your virtual environment, you need to issue the deactivate command from anywhere on the system:

deactivate

Your prompt should revert to the conventional display. When you wish to work on your project again, re-activate your virtual environment by moving back into your project directory and activating:

cd ~/newproject
source my_env/bin/activate

Development Version Install with Git

If you need a development version of Django, you can download and install Django from its Git repository. Let's do this from within a virtual environment.

First, let's update the local package index:

sudo apt update

Check the version of Python you have installed:

python3 -V

Output
Python 3.6.5

Next, install pip from the official repositories:

sudo apt install python3-pip

Install the venv package to create your virtual environment:

sudo apt install python3-venv

The next step is cloning the Django repository. Between releases, this repository will have more up-to-date features and bug fixes at the possible expense of stability. You can clone the repository to a directory called ~/django-dev within your home directory by typing:

git clone git://github.com/django/django ~/django-dev

Change to this directory:

cd ~/django-dev

Create a virtual environment using the python command that's compatible with your installed version of Python:

python3.6 -m venv my_env

Activate it:

source my_env/bin/activate

Next, you can install the repository using pip. The -e option will install in "editable" mode, which is necessary when installing from version control:

pip install -e ~/django-dev

You can verify that the installation was successful by typing:

django-admin --version

Output
2.2.dev20180802155335

Again, the version you see displayed may not match what is shown here.

You now have the latest version of Django in your virtual environment.

Creating a Sample Project

With Django installed, you can begin building your project. We will go over how to create a project and test it on your development server using a virtual environment.

First, create a directory for your project and change into it:

mkdir ~/django-test
cd ~/django-test

Next, create your virtual environment:

python3.6 -m venv my_env

Activate the environment:

source my_env/bin/activate

Install Django:

pip install django

To build your project, you can use django-admin with the startproject command. We will call our project djangoproject, but you can replace this with a different name. startproject will create a directory within your current working directory that includes:

A management script, manage.py, which you can use to administer various Django-specific tasks.
A directory (with the same name as the project) that includes the actual project code.

To avoid having too many nested directories, however, let's tell Django to place the management script and inner directory in the current directory (notice the ending dot):

django-admin startproject djangoproject .

To migrate the database (this example uses SQLite by default), let's use the migrate command with manage.py. Migrations apply any changes you've made to your Django models to your database schema.

To migrate the database, type:

python manage.py migrate

You will see output like the following:

OutputOperations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

Finally, let's create an administrative user so that you can use the Djano admin interface. Let's do this with the createsuperuser command:

python manage.py createsuperuser

You will be prompted for a username, an email address, and a password for your user.

Modifying ALLOWED_HOSTS in the Django Settings

To successfully test your application, you will need to modify one of the directives in the Django settings.

Open the settings file by typing:

nano ~/django-test/djangoproject/settings.py

Inside, locate the ALLOWED_HOSTS directive. This defines a whitelist of addresses or domain names that may be used to connect to the Django instance. An incoming request with a Host header that is not in this list will raise an exception. Django requires that you set this to prevent a certain class of security vulnerability.

In the square brackets, list the IP addresses or domain names that are associated with your Django server. Each item should be listed in quotations, with separate entries separated by a comma. If you want requests for an entire domain and any subdomains, prepend a period to the beginning of the entry:

~/django-test/djangoproject/settings.py

. . .
ALLOWED_HOSTS = ['your_server_ip_or_domain', 'your_second_ip_or_domain', . . .]

When you are finished, save the file and exit your editor.

Testing the Development Server

Once you have a user, you can start up the Django development server to see what a fresh Django project looks like. You should only use this for development purposes. When you are ready to deploy, be sure to follow Django's guidelines on deployment carefully.

Before you try the development server, make sure you open the appropriate port in your firewall. If you followed the initial server setup guide and are using UFW, you can open port 8000 by typing:

sudo ufw allow 8000

Start the development server:

python manage.py runserver your_server_ip:8000

Visit your server's IP address followed by :8000 in your web browser:

http://your_server_ip:8000

You should see something that looks like this:

To access the admin interface, add /admin/ to the end of your URL:

http://your_server_ip:8000/admin/

This will take you to a log in screen:

If you enter the admin username and password that you just created, you will have access to the main admin section of the site:

For more information about working with the Django admin interface, please see "How To Enable and Connect the Django Admin Interface."

When you are finished looking through the default site, you can stop the development server by typing CTRL-C in your terminal.

The Django project you've created provides the structural basis for designing a more complete site. Check out the Django documentation for more information about how to build your applications and customize your site.

Conclusion

You should now have Django installed on your Ubuntu 18.04 server, providing the main tools you need to create powerful web applications. You should also know how to start a new project and launch the developer server. Leveraging a complete web framework like Django can help make development faster, allowing you to concentrate only on the unique aspects of your applications.

If you would like more information about working with Django, including in-depth discussions of things like models and views, please see our Django development series.

Как установить веб-сервер Apache в Ubuntu 18.04

2018-08-03T21:25:34Z

Введение

HTTP сервер Apache является самым широко используемым веб-сервером в мире. Он предоставляет множество удобных функций включая динамически загружаемые модули, широкую поддержку мультимедиа, и интеграцию с другим популярным программным обеспечением.

В этом руководстве мы расскажем, как установить веб-сервер Apache на ваш сервер с Ubuntu 18.04.

Необходимые условия

Перед тем, как начать следовать шагам, описанным в этом руководстве, вам необходимо настроить отдельный, не-рутовый (non-root) профиль пользователя на вашем сервере с Ubuntu 18.04. Кроме того, вам потребуется настроить базовый файрвол для блокирования всех портов, кроме необходимых для работы Apache. Вы можете ознакомиться с процессом настройки аккаунта пользователя и настройкой файрвола на вашем сервере следуя шагам нашего руководства по первичной настройке сервера на Ubuntu 18.04.

После завершения создания аккаунта войдите на ваш сервер с помощью вновь созданного пользователя.

Шаг 1 - Установка Apache

Apache доступен из дефолтных репозиториев Ubuntu, что позволяет устанавливать его с помощью средств управления пакетами.

Давайте начнём с обновления локального индекса пакетов:

sudo apt update

Далее установим пакет apache2:

sudo apt install apache2

После подтверждения установки apt установит Apache и все необходимые зависимости.

Шаг 2 - Настройка файрвола

Перед тестированием установки Apache необходимо изменить настройки файрвола для разрешения доступа извне к дефолным веб-портам. Если вы следовали инструкциям по настройке файрвола из руководства по первичной настройке сервера, ваш файрвол UFW уже должен быть настроен таким образом, чтобы ограничивать доступ к вашему серверу.

В процессе установки Apache регистрирует себя в конфигурации UFW, создавая несколько профилей приложения, которые могут быть использованы для включения и отключения доступа к Apache через файрвол.

Выведем профили приложений ufw следующей командой:

sudo ufw app list

Вы увидите список приложений пользователей:

ВыводAvailable applications:
  Apache
  Apache Full
  Apache Secure
  OpenSSH

Как видно из этого вывода, для Apache доступно три профиля:

Apache: этот профиль открывает порт 80 (обычный, не шифрованный веб-трафик).
Apache Full: этот профиль открывает порты 80 (обычный, не шифрованный веб-трафик) и 443 (трафик шифруется с помощью TLS/SSL).
Apache Secure: этот профиль открывает только порт 443 (трафик шифруется с помощью TLS/SSL).

Рекомендуется включать самый ограниченный профиль, который будет позволять входящий трафик. Поскольку мы не настраивали SSL для нашего сервера в этом руководстве, нам потребуется включить только порт 80:

sudo ufw allow 'Apache'

Вы можете проверить внесённые изменения командой:

sudo ufw status

В выводе вы должны видеть, что HTTP трафик разрешён:

ВыводStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Apache                     ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Apache (v6)                ALLOW       Anywhere (v6)

Как видно из этого вывода профиль был включен для разрешения доступа к веб-серверу.

Шаг 3 - Проверка вашего веб-сервера

После завершения процесса установки Ubuntu 18.04 запустит Apache. Веб-сервер уже должен быть запущен.

Проверим в системе инициализации systemd, что сервис работает, следующей командой:

sudo systemctl status apache2

Вывод● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Tue 2018-04-24 20:14:39 UTC; 9min ago
 Main PID: 2583 (apache2)
    Tasks: 55 (limit: 1153)
   CGroup: /system.slice/apache2.service
           ├─2583 /usr/sbin/apache2 -k start
           ├─2585 /usr/sbin/apache2 -k start
           └─2586 /usr/sbin/apache2 -k start

Как видно из представленного вывода, сервис выглядит работающим корректно. Тем не менее, самый надёжный способ проверить работу Apache - это запросить веб-страницу.

Вы можете запросить дефолтную веб-страницу Apache с помощью IP адреса вашего сервера. Если вы не знаете IP адрес вашего сервера, вы можете найти его несколькими способами с помощью командной строки.

Введите следующую команду:

hostname -I

Она вернёт несколько адресов, разделённых пробелами. Вы можете попробовать каждый из них в вашем веб-браузере.

Другой способ заключается в использовании команды, которая позволяет увидеть ваш IP адрес из другого места в сети Интернет:

curl -4 icanhazip.com

После того, как вы найдёте IP адрес вашего сервера, введите его в свой веб-браузер:

http://IP_адрес_вашего_сервера

Вы должны увидеть дефолтную страницу Apache для Ubuntu 18.04:

Эта страница свидетельствует о том, что Apache работает корректно. На этой странице также представлена базовая информация о важных файлах и директориях Apache.

Шаг 4 - Управление процессом Apache

Теперь, когда у вас есть работающий веб-сервер, рассмотрим некоторые базовые команды для управления им.

Для остановки себ-сервера наберите:

sudo systemctl stop apache2

Для запуска остановленного сервера наберите:

sudo systemctl start apache2

Для перезапуска сервиса наберите:

sudo systemctl restart apache2

Если вы вносите какие-то изменения в конфигурацию, Apache зачастую может перезагружаться без потери открытых соединений. Для этого наберите команду:

sudo systemctl reload apache2

По умолчанию Apache сконфигурирован на запуск при загрузке сервера. Вы можете отключить такое поведение следующей командой:

sudo systemctl disable apache2

Для повторного включения сервиса при загрузке сервера наберите:

sudo systemctl enable apache2

Теперь Apache должен опять запускаться автоматически при загрузке сервера.

Шаг 5 - Настройка виртуальных хостов (рекомендуется)

При использовании веб-сервера Apache вы можете использовать виртуальные хосты (аналог серверных блоков в Nginx) для хранения конфигурационных настроек разных сайтов. Это позволяет иметь более одного сайта на одном сервере. В этом руководстве мы будем для примера использовать доменное имя example.com, но вам следует заменить его вашим собственным доменным именем. Для того, чтобы узнать больше о настройке доменных имён в DigitalOcean, рекомендуем ознакомиться с нашим Введением в DNS DigitalOcean.

Apache для Ubuntu 18.04 уже имеет один виртуальный хост, включенный по умолчанию, который настроен на отдачу документов из директории /var/www/html. Хотя это и удобно для обслуживания одного сайта, это становится неудобным, когда сайтов несколько. Вместо того, чтобы изменять /var/www/html, давайте создадим новую структуру директорий внутри /var/www для нашего сайта example.com, оставив /var/www/html для показа дефолтной страницы пользователям в случаях, когда клиентский запрос не совпадает ни с одним из настроенных доменных имён.

Создайте директорию для example.com используя флаг -p для создания необходимых родительских директорий:

sudo mkdir -p /var/www/example.com/html

Далее настройте владельца директории с помощью переменной окружения $USER:

sudo chown -R $USER:$USER /var/www/example.com/html

Теперь права должны для корневой директории быть настроены правильным образом при условии, что вы не меняли своё значение umask. На всякий случай мы можем удостовериться в этом командой:

sudo chmod -R 755 /var/www/example.com

Далее создадим страницу index.html в nano или любом другом текстовом редакторе:

nano /var/www/example.com/html/index.html

Добавим в файл следующий HTML:

/var/www/example.com/html/index.html

<html>
    <head>
        <title>Welcome to Example.com!</title>
    </head>
    <body>
        <h1>Success!  The example.com server block is working!</h1>
    </body>
</html>

Сохраните и закройте файл.

Для того, чтобы Apache мог отдавать этот контент, нам необходимо настроить виртуальный хост с корректными настройками. Вместо того, чтобы редактировать существующий файл виртуального хоста /etc/apache2/sites-available/000-default.conf, создадим новый файл для нашего сайта - /etc/apache2/sites-available/example.com.conf:

sudo nano /etc/apache2/sites-available/example.com.conf

Скопируйте следующий текст настроек виртуального хоста в созданный файл:

/etc/apache2/sites-available/example.com.conf

<VirtualHost *:80>
    ServerAdmin admin@example.com
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/example.com/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Обратите внимание, что мы обновили DocumentRoot на адрес нашей новой директории, и ServerAdmin на адрес электронной почты, доступный для администратора example.com. Мы также добавили две директивы: ServerName, которая устанавливает базовое доменное имя, которое должно использоваться для хоста, а также ServerAlias, которая определяет другие имена, которые должны использоваться для отображения хоста так же, как и базовое доменное имя.

Сохраните и закройте файл после внесения изменений.

Теперь активируем профиль сайта с помощью утилиты a2ensite:

sudo a2ensite example.com.conf

Деактивируем дефолтный сайт, определённый в 000-default.conf:

sudo a2dissite 000-default.conf

Далее проверим наши настройки на наличие ошибок:

sudo apache2ctl configtest

Вы должны увидеть следующий вывод:

ВыводSyntax OK

Перезапустите Apache для применения внесённых изменений:

sudo systemctl restart apache2

Теперь Apache должен работать с вашим доменным именем. Вы можете проверить это введя http://example.com в вашем браузере, где в результате вы должны увидеть что-то в этом роде:

Шаг 6 - Важные файлы и директории Apache

Теперь, когда вы знаете, как управлять сервисом Apache, вам стоит ознакомиться с важными файлами и директориями Apache.

Контент

/var/www/html: фактический веб-контент, который по умолчанию состоит только из дефолтной страницы Apache, которую мы видели ранее, хранится в директории /var/www/html. Это может быть изменено в конфигурационных файлах Apache.

Конфигурация сервера

/etc/apache2: это конфигурационная директория Apache. Все файлы конфигурации Apache находятся здесь.
/etc/apache2/apache2.conf: главный конфигурационный файл Apache. Изменения в этом файле влияют на глобальную конфигурацию Apache. Этот файл отвечает за загрузку многих других файлов из конфигурационной директории.
/etc/apache2/ports.conf: этот файл определяет порты, которые Apache будет слушать. По умолчанию Apache слушает порт 80, а также порт 443 при условии, что модуль для работы с SSL включен.
/etc/apache2/sites-available/: в этой директории хранятся файлы виртуальных хостов. Apache не использует файлы из этой директории, если ссылки на них нет в директории sites-enabled. Обычно настройка всех файлов виртуальных хостов осуществляется в этой директории, а активация хоста происходит путём создания ссылки в другой директории командой a2ensite.
/etc/apache2/sites-enabled/: директория, в которой хранятся активированные виртуальные хосты. Обычно это делается путём создания ссылки на файл конфигурации хоста из директории sites-available с помощью команды a2ensite. Apache читает конфигурационный файлы и ссылки из этой директории при запуске или перезапуске.
/etc/apache2/conf-available/, /etc/apache2/conf-enabled/: эти директории связаны друг с другом так же, как и sites-available и sites-enabled связаны друг с другом, но используются для хранения фрагментов конфигурации, которые не принадлежат виртуальным хостам. Файлы в директории conf-available могут быть включены командой a2enconf и выключены командой a2disconf.
/etc/apache2/mods-available/, /etc/apache2/mods-enabled/: эти директории содержат, соответственно, доступные и активные модули. Файлы, оканчивающиеся на .load, содержат фрагменты для загрузки конкретных модулей, а файлы, оканчивающиеся на .conf, содержат настройки этих модулей. Модули можно активировать командой a2enmod и деактивировать командой a2dismod.

Серверные логи

/var/log/apache2/access.log: по умолчанию каждый запрос к вашему веб-серверу записывается в этом файле, если только Apache не настроен на другое поведение.
/var/log/apache2/error.log: по умолчанию все ошибки записываются в этот файл. Директива LogLevel в конфигурации Apache определяет, насколько детальными должны быть записи об ошибках.

Заключение

Теперь, когда ваш веб-сервер установлен, у вас есть множество вариантов того, что делать дальше. Если вы хотите построить более полный стек приложений, вы можете ознакомиться с нашим руководством по установке и настройке стека LAMP на Ubuntu 18.04.

Как повысить безопасность Apache с помощью Let's Encrypt в Ubuntu 18.04

2018-08-03T21:17:59Z

Введение

Let's Encrypt представляет собой центр сертификации (Certificate Authority, CA), позволяющий получать и устанавливать бесплатные сертификаты TLS/SSL, тем самым позволяя использовать шифрованный HTTPS на веб-серверах. Процесс получения сертификатов упрощается за счёт наличия клиента Certbot, который пытается автоматизировать большую часть (если не все) необходимых операций. В настоящее время весь процесс получения и установки сертификатов полностью автоматизирован и для Apache и для Nginx.

В этом руководстве мы используем Certbot для получения бесплатного SSL сертификата для Apache на Ubuntu 18.04, а также настроим автоматическое продление этого сертификата.

В этом руководстве мы будем использовать файл отдельного виртуального хоста Apache вместо дефолтного файла конфигурации. Мы рекомендуем создавать новый файлы виртуальных хостов Apache для каждого доменного имени, потому что это помогает избегать распространённых ошибок и использовать дефолтные файлы в качестве примера корректной конфигурации, когда что-нибудь пойдёт не так.

Перед установкой

Перед тем, как начать следовать описанным в этой статье шагам, убедитесь, что у вас есть:

Сервер с Ubuntu 18.04, настроенный согласно руководству по первичной настройке сервера с Ubuntu 18.04, включая настройку не-рутового (non-root) пользователя с привилегиями sudo и настройку файрвола.
Зарегистрированное доменное имя. В этом руководстве мы будем использовать example.com. Вы можете приобрести доменное имя на Namecheap, получить бесплатное доменное имя на Freenom или использовать любой другой регистратор доменных имён.
Для вашего сервера настроены обе записи DNS, указанные ниже. Для их настройки вы можете использовать наше введение в работу с DNS в DigitalOcean.
- Запись A для example.com, указывающая на публичный IP адрес вашего сервера.
- Запись A для www.example.com, указывающая на публичный IP адрес вашего сервера.
Apache, установленный согласно инструкциям из руководства Как установить Apache в Ubuntu 18.04. Убедитесь, что у вас есть настроенный файл виртуального хоста для вашего домена. В этом руководстве мы будем использовать /etc/apache2/sites-available/example.com.conf в качестве примера.

Шаг 1 - Установка Certbot

Перед началом использования Let's Encrypt для получения SSL сертификаты установим Certbot на ваш сервер.

Certbot находится в активной разработке, поэтому пакеты Certbot, предоставляемые Ubuntu, обычно являются устаревшими. Тем не менее, разработчики Certbot поддерживают свой репозиторий пакетов для Ubuntu с актуальными версиями, поэтому мы будем использовать именно этот репозиторий.

Сначала добавим репозиторий:

sudo add-apt-repository ppa:certbot/certbot

Далее нажмите ENTER.

Установим пакет Certbot для Apache с помощью apt:

sudo apt install python-certbot-apache

Теперь Certbot готов к использованию, но для того, чтобы он мог настроить SSL для Apache, нам сперва необходимо проверить кое-какие настройки Apache.

Шаг 2 - Настройка SSL сертификата

Certbot должен иметь возможность найти корректный виртуальный хост в вашей конфигурации Apache для того, чтобы автоматически конфигурировать SSL. Для этого он будет искать директиву ServerName, которая совпадает с доменным именем, для которого вы запросите сертификат.

Если вы следовали инструкциям по настройке виртуального хоста в руководстве по установке Apache, у вас должен быть виртуальный хост для вашего домена по адресу /etc/apache2/sites-available/example.com.conf с уже правильно настроенной директивой ServerName.

Для проверки откройте файл серверного блока в nano или любом другом текстовом редакторе:

sudo nano /etc/apache2/sites-available/example.com.conf

Найдите строку с ServerName. Она должна выглядеть примерно так:

/etc/apache2/sites-available/example.com.conf

...
ServerName example.com;
...

Если она выглядит таким образом, закройте файл и переходите к следующему шагу.

Если она не выглядит так, как описано выше, обновите директиву ServerName. Затем сохраните и закройте файл, после чего проверьте корректность синтаксиса вашего конфигурационного файла командой:

sudo apache2ctl configtest

Если вы получили ошибку, откройте файл серверного блока и проверьте его на наличие опечаток или пропущенных символов. После того, как ваш конфигурационный файл будет проходить проверку на корректность, перезагрузите Apache для применения новой конфигурации:

sudo systemctl reload apache2

Теперь Certbot может находить и обновлять корректный виртуальный хост.

Далее обновим настройки файрвола для пропуска HTTPS трафика.

Шаг 3 - Разрешение HTTPS в файрволе

Если у вас включен файрвол ufw, как рекомендуется в руководстве по первичной настройке сервера, вам необходимо внести некоторые изменения в его настройки для разрешения трафика HTTPS. К счастью, Apache регистрирует необходимые профили в ufw в момент установки.

Вы можете ознакомиться с текущими настройками командой:

sudo ufw status

Скорее всего вывод будет выглядеть следующим образом:

ВыводStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Apache                     ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Apache (v6)                ALLOW       Anywhere (v6)

Как видно из вывода, разрешён только трафик HTTP.

Для того, чтобы разрешить трафик HTTPS, разрешим профиль Apache Full и удалим избыточный профиль Apache:

sudo ufw allow 'Apache Full'
sudo ufw delete allow 'Apache'

Проверим внесённые изменения:

sudo ufw status

Теперь настройки ufw должны выглядеть следующим образом:

ВыводStatus: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Apache Full                ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Apache Full (v6)           ALLOW       Anywhere (v6)

Теперь мы можем запустить Certbot и получить наши сертификаты.

Шаг 4 - Получение SSL сертификата

Certbot предоставляет несколько способов получения сертификатов SSL с использованием плагинов. Плагин для Apache берёт на себя настройку Apache и перезагрузку конфигурации, когда это необходимо. Для использования плагина выполним команду:

sudo certbot --apache -d example.com -d www.example.com

Эта команда запускает certbot с плагином --apache, ключи -d определяют имена доменов, для которых должен быть выпущен сертификат.

Если это первый раз, когда вы запускаете certbot, вам будет предложено ввести адрес электронной почты и согласиться с условиями использования сервиса. После этого certbot свяжется с сервером Let's Encrypt, а затем проверит, что вы действительно контролируете домен, для которого вы запросили сертификат.

Если всё прошло успешно, certbot спросит, как вы хотите настроить конфигурацию HTTPS.

ВыводPlease choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Выберите подходящий вариант и нажмите ENTER. Конфигурация будет обновлена, а Apache перезапущен для применения изменений. certbot выдаст сообщение о том, что процесс прошёл успешно, и где хранятся ваши сертификаты:

ВыводIMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2018-07-23. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Ваши сертификаты загружены, установлены и работают. Попробуйте перезагрузить ваш сайт с использованием https:// и вы увидите значок безопасности в браузере. Он означает, что соединение с сайтом зашифровано, обычно он выглядит, как зелёная иконка замка. Если вы проверите ваш сервер тестом SSL Labs Server Test, он получит оценку A.

Закончим тестированием процесса обновления сертификата.

Шаг 5 - Проверка автоматического обновления сертификата

Сертификаты Let's Encrypt действительны только 90 дней. Это сделано для того, чтобы пользователи автоматизировали процесс обновления сертификатов. Пакет certbot, который мы установили, делает это путём добавления скрипта обновления в /etc/cron.d. Этот скрипт запускается раз в день и автоматически обновляет любые сертификаты, которые закончатся в течение ближайших 30 дней.

Для тестирования процесса обновления мы можем сделать "сухой" запуск (dry run) certbot:

sudo certbot renew --dry-run

Если вы не видите каких-либо ошибок в результате выполнения этой команды, то всё в полном порядке. При необходимости Certbot будет обновлять ваши сертификаты и перезагружать Apache для применения изменений. Если автоматическое обновление по какой-либо причине закончится ошибкой, Let's Encrypt отправит электронное письмо на указанный вами адрес электронной почты с информацией о сертификате, который скоро закончится.

Заключение

В этом руководстве мы рассмотрели процесс установки клиента Let's Encrypt certbot, загрузили SSL сертификаты для вашего домена, настроили Apache для использования этих сертификатов и настроили процесс автоматического обновления сертификатов. Если у вас есть вопросы по работе с Certbot, рекомендуем ознакомиться с документацией Certbot.

DigitalOcean Community Tutorials

An Introduction to the Kubernetes DNS Service

Introduction

What Does the Kubernetes DNS Service Provide?

Example Kubernetes DNS Records

Search Domains and Resolving Shorter Hostnames

Kubernetes DNS Implementation Details

kube-dns

CoreDNS

Additional Configuration Options

Conclusion

How to Install and Configure pgAdmin 4 in Server Mode

Introduction

Prerequisites

Step 1 — Installing pgAdmin and its Dependencies

Step 2 — Configuring pgAdmin 4

Step 3 — Configuring Apache

Step 4 — Accessing pgAdmin

Step 5 — Configuring your PostgreSQL User

Step 6 — Creating a Table in the pgAdmin Dashboard

Conclusion

How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image

Introduction

Prerequisites

Step 1 — Installing VirtualBox and Creating a Virtual Machine

Step 2 — Installing Ubuntu 18.04 onto the Virtual Machine

Step 3 — Reconfiguring cloud-init

Step 4 — Uploading Custom Image and Creating Droplet

Conclusion

How to Deploy a Symfony 4 Application to Production with LEMP on Ubuntu 18.04

Introduction

Prerequisites

Step 1 — Creating a User and Database for the Application

Step 2 — Setting Up the Demo Application

Step 3 — Configuring your Environment Variables for the Application

Step 4 — Setting Up Database Credentials

Step 5 — Populating your Database Using Doctrine-Fixtures

Step 6 — Clearing and Warming Up your Cache

Step 7 — Configuring the Web Server and Running the Application

Conclusion

An Introduction to Queries in MySQL

Introduction

Prerequisites

Creating a Sample Database

Understanding SELECT Statements

Aggregate Functions

Manipulating Query Outputs

Querying Multiple Tables

Conclusion

An Introduction to Queries in PostgreSQL

Introduction

Prerequisites

Creating a Sample Database

Understanding SELECT Statements

Aggregate Functions

Manipulating Query Outputs

Querying Multiple Tables

Conclusion

Usando uma CDN para Acelerar a Entrega de Conteúdo Estático

Introdução

O que é uma CDN?

Como uma CDN funciona?

Zonas Push versus Zonas Pull

Benefícios do Uso de uma CDN

Descarregamento da Origem

Latência Mais Baixa para uma Melhor Experiência do Usuário

Gerenciar Picos de Tráfego e Evitar Tempo de Inatividade

Reduzir Custos

Aumentar a Segurança

Escolhendo a Melhor Solução

Conclusão

Como criar um Space e uma API Key na DigitalOcean

Introdução

Pré-requisitos

Criando um Space

Criando uma Access Key

Conclusão

Como Configurar Pipelines de Integração Contínua com o GitLab CI no Ubuntu 16.04

Introdução

Pré-requisitos

Step 3 — Reconfiguring `cloud-init`