One place for hosting & domains

      October 2018

      How To Test Ansible Roles with Molecule on Ubuntu 18.04


      The author selected the Mozilla Foundation to receive a donation as part of the Write for DOnations program.

      Introduction

      Unit testing in Ansible is key to making sure roles function as intended. Molecule makes this process easier by allowing you to specify scenarios that test roles against different environments. Using Ansible under the hood, Molecule offloads roles to a provisioner that deploys the role in a configured environment and calls a verifier (such as Testinfra) to check for configuration drift. This ensures that your role has made all of the expected changes to the environment in that particular scenario.

      In this guide, you will build an Ansible role that deploys Apache to a host and configures firewalld on CentOS 7. To test that this role works as intended, you will create a test in Molecule using Docker as a driver and Testinfra, a Python library for testing the state of servers. Molecule will provision Docker containers to test the role and Testinfra will verify that the server has been configured as intended. When you’re finished, you’ll be able to create multiple test cases for builds across environments and run these tests using Molecule.

      Prerequisites

      Before you begin this guide you’ll need the following:

      Step 1 — Preparing the Environment

      If you’ve followed the prerequisites, you should have Python 3, venv, and Docker installed and correctly configured. Let’s begin by creating a virtual environment to test Ansible with Molecule.

      Start by logging in as your non-root user and creating a new virtual environment:

      Activate it to ensure that your actions are restricted to that environment:

      • source my_env/bin/activate

      Next, in your activated environment, install the wheel package, which provides the bdist_wheel setuptools extension that pip uses to install Ansible:

      • python3 -m pip install wheel

      You can now install molecule and docker with pip. Ansible will be automatically installed as a dependency for Molecule:

      • python3 -m pip install molecule docker

      Here is what each of these packages will do:

      • molecule: This is the main Molecule package that you will use to test roles. Installing molecule automatically installs Ansible, along with other dependencies, and enables the use of Ansible playbooks to execute roles and tests.
      • docker: This Python library is used by Molecule to interface with Docker. You will need this since you're using Docker as a driver.

      Next, let's create a role in Molecule.

      Step 2 — Creating a Role in Molecule

      With your environment set up, you can use Molecule to create a basic role that you will use to test an installation of Apache. This role will create the directory structure and some initial tests, and specify Docker as the driver so that Molecule uses Docker to run its tests.

      Create a new role called ansible-apache:

      • molecule init role -r ansible-apache -d docker

      The -r flag specifies the name of the role while -d specifies the driver, which provisions the hosts for Molecule to use in testing.

      Change into the directory of the newly created role:

      Test the default role to check if Molecule has been set up properly:

      You will see output that lists each of the default test actions. Before starting the test, Molecule validates the configuration file molecule.yml to make sure everything is in order. It also prints this test matrix, which specifies the order of test actions:

      Output

      --> Validating schema /home/sammy/ansible-apache/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix └── default ├── lint ├── destroy ├── dependency ├── syntax ├── create ├── prepare ├── converge ├── idempotence ├── side_effect ├── verify └── destroy ...

      We will discuss each test action in detail once you've created your role and customized your tests. For now, pay attention to the PLAY_RECAP for each test, and be sure that none of the default actions returns a failed status. For example, the PLAY_RECAP for the default 'create' action should look like this:

      Output

      ... PLAY RECAP ********************************************************************* localhost : ok=5 changed=4 unreachable=0 failed=0

      Let's move on to modifying the role to configure Apache and firewalld.

      Step 3 — Configuring Apache and Firewalld

      To configure Apache and firewalld, you will create a tasks file for the role, specifying packages to install and services to enable. These details will be extracted from a variables file and template that you will use to replace the default Apache index page.

      Still in the ansible-apache directory, create a tasks file for the role using nano or your favorite text editor:

      You'll see that the file already exists. Delete what's there and replace it with the following code to install the required packages and enable the correct services, HTML defaults, and firewall settings:

      ~/ansible-apache/tasks/main.yml

      ---
      - name: "Ensure required packages are present"
        yum:
          name: "{{ pkg_list }}"
          state: present
      
      - name: "Ensure latest index.html is present"
        template:
          src: index.html.j2
          dest: /var/www/html/index.html
      
      - name: "Ensure httpd service is started and enabled"
        service:
          name: "{{ item }}"
          state: started
          enabled: true
        with_items: "{{ svc_list }}"
      
      - name: "Whitelist http in firewalld"
        firewalld:
          service: http
          state: enabled
          permanent: true
          immediate: true
      

      This playbook includes 4 tasks:

      • "Ensure required packages are present": This task will install the packages listed in the variables file under pkg_list. The variables file will be located at ~/ansible-apache/vars/main.yml and you will create it at the end of this step.
      • "Ensure latest index.html is present": This task will copy a template page, index.html.j2, and paste it over the default index file, /var/www/html/index.html, generated by Apache. You will also create the new template in this step.
      • "Ensure httpd service is started and enabled": This task will start and enable the services listed in svc_list in the variables file.
      • "Whitelist http in firewalld": This task will whitelist the http service in firewalld. Firewalld is a complete firewall solution present by default on CentOS servers. For the http service to work, you will need to expose the required ports. Instructing firewalld to whitelist a service ensures that it whitelists all of the ports that the service requires.

      Save and close the file when you are finished.

      Next, let's create a templates directory for the index.html.j2 template page:

      Create the page itself:

      • nano templates/index.html.j2

      Paste in the following boilerplate code:

      ~/ansible-apache/templates/index.html.j2

      <div style="text-align: center">
          <h2>Managed by Ansible</h2>
      </div>
      

      Save and close the file.

      The final step in completing the role is writing the variables file, which provides the names of packages and services to our main role playbook:

      Paste over the default content with the following code, which specifies pkg_list and svc_list:

      ~/ansible-apache/vars/main.yml

      ---
      pkg_list:
        - httpd
        - firewalld
      svc_list:
        - httpd
        - firewalld
      

      These lists contain the following information:

      • pkg_list: This contains the names of the packages that the role will install: httpd and firewalld.
      • svc_list: This contains the names of the services that the role will start and enable: httpd and firewalld.

      Note: Make sure that your variables file doesn't have any blank lines or your test will fail during linting.

      Now that you've finished creating your role, let's configure Molecule to test if it works as intended.

      Step 4 — Modifying the Role for Running Tests

      In our case, configuring Molecule involves modifying the Molecule configuration file molecule.yml to add platform specifications. Because you're testing a role that configures and starts the httpd systemd service, you will need to use an image with systemd configured and privileged mode enabled. For this tutorial, you will use the milcom/centos7-systemd image available on Docker Hub. Privileged mode allows containers to run with almost all of the capabilities of their host machine.

      Let's edit molecule.yml to reflect these changes:

      • nano molecule/default/molecule.yml

      Add the highlighted platform information:

      ~/ansible-apache/molecule/default/molecule.yml

      ---
      dependency:
        name: galaxy
      driver:
        name: docker
      lint:
        name: yamllint
      platforms:
        - name: centos7
          image: milcom/centos7-systemd
          privileged: true
      provisioner:
        name: ansible
        lint:
          name: ansible-lint
      scenario:
        name: default
      verifier:
        name: testinfra
        lint:
          name: flake8
      

      Save and close the file when you are done.

      Now that you've successfully configured the test environment, let's move on to writing the test cases that Molecule will run against your container after executing the role.

      Step 5 — Writing Test Cases

      In the test for this role, you will check the following conditions:

      • That the httpd and firewalld packages are installed.
      • That the httpd and firewalld services are running and enabled.
      • That the http service is enabled in your firewall settings.
      • That index.html contains the same data specified in your template file.

      If all of these tests pass, then the role works as intended.

      To write the test cases for these conditions, let's edit the default tests in ~/ansible-apache/molecule/default/tests/test_default.py. Using Testinfra, we will write the test cases as Python functions that use Molecule classes.

      Open test_default.py:

      • nano molecule/default/tests/test_default.py

      Delete the contents of the file so that you can write the tests from scratch.

      Note: As you write your tests, make sure that they are separated by two new lines or they will fail.

      Start by importing the required Python modules:

      ~/ansible-apache/molecule/default/tests/test_default.py

      import os
      import pytest
      
      import testinfra.utils.ansible_runner
      

      These modules include:

      • os: This built-in Python module enables operating-system-dependent functionality, making it possible for Python to interface with the underlying operating system.
      • pytest: The pytest module enables test writing.
      • testinfra.utils.ansible_runner: This Testinfra module uses Ansible as the backend for command execution.

      Under the module imports, add the following code, which uses the Ansible backend to return the current host instance:

      ~/ansible-apache/molecule/default/tests/test_default.py

      ...
      testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
          os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
      

      With your test file configured to use the Ansible backend, let's write unit tests to test the state of the host.

      The first test will ensure that httpd and firewalld are installed:

      ~/ansible-apache/molecule/default/tests/test_default.py

      ...
      
      @pytest.mark.parametrize('pkg', [
        'httpd',
        'firewalld'
      ])
      def test_pkg(host, pkg):
          package = host.package(pkg)
      
          assert package.is_installed
      

      The test begins with the pytest.mark.parametrize decorator, which allows us to parameterize the arguments for the test. This first test will take test_pkg as a parameter to test for the presence of the httpd and firewalld packages.

      The next test checks whether or not httpd and firewalld are running and enabled. It takes test_svc as a parameter:

      ~/ansible-apache/molecule/default/tests/test_default.py

      ...
      
      @pytest.mark.parametrize('svc', [
        'httpd',
        'firewalld'
      ])
      def test_svc(host, svc):
          service = host.service(svc)
      
          assert service.is_running
          assert service.is_enabled
      

      The last test checks that the files and contents passed to parametrize() exist. If the file isn't created by your role and the content isn't set properly, assert will return False:

      ~/ansible-apache/molecule/default/tests/test_default.py

      ...
      
      @pytest.mark.parametrize('file, content', [
        ("/etc/firewalld/zones/public.xml", "<service name="http"/>"),
        ("/var/www/html/index.html", "Managed by Ansible")
      ])
      def test_files(host, file, content):
          file = host.file(file)
      
          assert file.exists
          assert file.contains(content)
      

      In each test, assert will return True or False depending on the test result.

      The finished file looks like this:

      ~/ansible-apache/molecule/default/tests/test_default.py

      import os
      import pytest
      
      import testinfra.utils.ansible_runner
      
      testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
          os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
      
      
      @pytest.mark.parametrize('pkg', [
        'httpd',
        'firewalld'
      ])
      def test_pkg(host, pkg):
          package = host.package(pkg)
      
          assert package.is_installed
      
      
      @pytest.mark.parametrize('svc', [
        'httpd',
        'firewalld'
      ])
      def test_svc(host, svc):
          service = host.service(svc)
      
          assert service.is_running
          assert service.is_enabled
      
      
      @pytest.mark.parametrize('file, content', [
        ("/etc/firewalld/zones/public.xml", "<service name="http"/>"),
        ("/var/www/html/index.html", "Managed by Ansible")
      ])
      def test_files(host, file, content):
          file = host.file(file)
      
          assert file.exists
          assert file.contains(content)
      

      Now that you've specified your test cases, let's test the role.

      Step 6 — Testing the Role with Molecule

      Once you initiate the test, Molecule will execute the actions you defined in your scenario. Let's now run the default molecule scenario again, executing the actions in the default test sequence while looking more closely at each.

      Run the test for the default scenario again:

      This will initiate the test run. The initial output prints the default test matrix:

      Output

      --> Validating schema /home/sammy/ansible-apache/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix └── default ├── lint ├── destroy ├── dependency ├── syntax ├── create ├── prepare ├── converge ├── idempotence ├── side_effect ├── verify └── destroy

      Let's go through each test action and the expected output, starting with linting.

      The linting action executes yamllint, flake8, and ansible-lint:

      • yamllint: This linter is executed on all YAML files present in the role directory.
      • flake8: This Python code linter checks tests created for Testinfra.
      • ansible-lint: This linter for Ansible playbooks is executed in all scenarios.

      Output

      ... --> Scenario: 'default' --> Action: 'lint' --> Executing Yamllint on files found in /home/sammy/ansible-apache/... Lint completed successfully. --> Executing Flake8 on files found in /home/sammy/ansible-apache/molecule/default/tests/... Lint completed successfully. --> Executing Ansible Lint on /home/sammy/ansible-apache/molecule/default/playbook.yml... Lint completed successfully.

      The next action, destroy, is executed using the destroy.yml file. This is done to test our role on a newly created container.

      By default, destroy is called twice: at the start of the test run, to delete any pre-existing containers, and at the end, to delete the newly created container:

      Output

      ... --> Scenario: 'default' --> Action: 'destroy' PLAY [Destroy] ***************************************************************** TASK [Destroy molecule instance(s)] ******************************************** changed: [localhost] => (item=None) changed: [localhost] TASK [Wait for instance(s) deletion to complete] ******************************* ok: [localhost] => (item=None) ok: [localhost] TASK [Delete docker network(s)] ************************************************ skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=2 changed=1 unreachable=0 failed=0

      After the destroy action is complete, the test will move on to dependency. This action allows you to pull dependencies from ansible-galaxy if your role requires them. In this case, our role does not:

      Output

      ... --> Scenario: 'default' --> Action: 'dependency' Skipping, missing the requirements file.

      The next test action is a syntax check, which is executed on the default playbook.yml playbook. It works in a similar way to the --syntax-check flag in the command ansible-playbook --syntax-check playbook.yml:

      Output

      ... --> Scenario: 'default' --> Action: 'syntax' playbook: /home/sammy/ansible-apache/molecule/default/playbook.yml

      Next, the test moves on to the create action. This uses the create.yml file in your role's Molecule directory to create a Docker container with your specifications:

      Output

      ... --> Scenario: 'default' --> Action: 'create' PLAY [Create] ****************************************************************** TASK [Log into a Docker registry] ********************************************** skipping: [localhost] => (item=None) skipping: [localhost] TASK [Create Dockerfiles from image names] ************************************* changed: [localhost] => (item=None) changed: [localhost] TASK [Discover local Docker images] ******************************************** ok: [localhost] => (item=None) ok: [localhost] TASK [Build an Ansible compatible image] *************************************** changed: [localhost] => (item=None) changed: [localhost] TASK [Create docker network(s)] ************************************************ skipping: [localhost] TASK [Create molecule instance(s)] ********************************************* changed: [localhost] => (item=None) changed: [localhost] TASK [Wait for instance(s) creation to complete] ******************************* changed: [localhost] => (item=None) changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=5 changed=4 unreachable=0 failed=0

      After create, the test moves on to the prepare action. This action executes the prepare playbook, which brings the host to a specific state before running converge. This is useful if your role requires a pre-configuration of the system before the role is executed. Again, this does not apply to our role:

      Output

      ... --> Scenario: 'default' --> Action: 'prepare' Skipping, prepare playbook not configured.

      After prepare, the converge action executes your role on the container by running the playbook.yml playbook. If multiple platforms are configured in the molecule.yml file, Molecule will converge on all of these:

      Output

      ... --> Scenario: 'default' --> Action: 'converge' PLAY [Converge] **************************************************************** TASK [Gathering Facts] ********************************************************* ok: [centos7] TASK [ansible-apache : Ensure required packages are present] ******************* changed: [centos7] TASK [ansible-apache : Ensure latest index.html is present] ******************** changed: [centos7] TASK [ansible-apache : Ensure httpd service is started and enabled] ************ changed: [centos7] => (item=httpd) changed: [centos7] => (item=firewalld) TASK [ansible-apache : Whitelist http in firewalld] **************************** changed: [centos7] PLAY RECAP ********************************************************************* centos7 : ok=5 changed=4 unreachable=0 failed=0

      After coverge, the test moves on to idempotence. This action tests the playbook for idempotence to make sure no unexpected changes are made in multiple runs:

      Output

      ... --> Scenario: 'default' --> Action: 'idempotence' Idempotence completed successfully.

      The next test action is the side-effect action. This lets you produce situations in which you'll be able to test more things, like HA failover. By default, Molecule doesn't configure a side-effect playbook and the task is skipped:

      Output

      ... --> Scenario: 'default' --> Action: 'side_effect' Skipping, side effect playbook not configured.

      Molecule will then run the verifier action using the default verifier, Testinfra. This action executes the tests you wrote earlier in test_default.py. If all the tests pass successfully, you will see a success message and Molecule will proceed to the next step:

      Output

      ... --> Scenario: 'default' --> Action: 'verify' --> Executing Testinfra tests found in /home/sammy/ansible-apache/molecule/default/tests/... ============================= test session starts ============================== platform linux -- Python 3.6.5, pytest-3.7.3, py-1.5.4, pluggy-0.7.1 rootdir: /home/sammy/ansible-apache/molecule/default, inifile: plugins: testinfra-1.14.1 collected 6 items tests/test_default.py ...... [100%] ========================== 6 passed in 41.05 seconds =========================== Verifier completed successfully.

      Finally, Molecule destroys the instances completed during the test and deletes the network assigned to those instances:

      Output

      ... --> Scenario: 'default' --> Action: 'destroy' PLAY [Destroy] ***************************************************************** TASK [Destroy molecule instance(s)] ******************************************** changed: [localhost] => (item=None) changed: [localhost] TASK [Wait for instance(s) deletion to complete] ******************************* changed: [localhost] => (item=None) changed: [localhost] TASK [Delete docker network(s)] ************************************************ skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=2 changed=2 unreachable=0 failed=0

      The test actions are now complete, verifying that your role worked as intended.

      Conclusion

      In this article you created an Ansible role to install and configure Apache and firewalld. You then wrote unit tests with Testinfra that Molecule used to assert that the role ran successfully.

      You can use the same basic method for highly complex roles, and automate testing using a CI pipeline as well. Molecule is a highly configurable tool that can be used to test roles with any providers that Ansible supports, not just Docker. It's also possible to automate testing against your own infrastructure, making sure that your roles are always up-to-date and functional. The official Molecule documentation is the best resource for learning how to use Molecule.



      Source link

      How To Provision and Manage Remote Docker Hosts with Docker Machine on Ubuntu 18.04


      Introduction

      [Docker Machine](/) is a tool that makes it easy to provision and manage multiple Docker hosts remotely from your personal computer. Such servers are commonly referred to as Dockerized hosts and are used to run Docker containers.

      While Docker Machine can be installed on a local or a remote system, the most common approach is to install it on your local computer (native installation or virtual machine) and use it to provision Dockerized remote servers.

      Though Docker Machine can be installed on most Linux distributions as well as macOS and Windows, in this tutorial, you’ll install it on your local machine running Ubuntu 18.04 and use it to provision Dockerized DigitalOcean Droplets. If you don’t have a local Ubuntu 18.04 machine, you can follow these instructions on any Ubuntu 18.04 server.

      Prerequisites

      To follow this tutorial, you will need the following:

      • A local machine or server running Ubuntu 18.04 with Docker installed. See How To Install and Use Docker on Ubuntu 18.04 for instructions.
      • A DigitalOcean API token. If you don’t have one, generate it using this guide. When you generate a token, be sure that it has read-write scope. That is the default, so if you do not change any options while generating it, it will have read-write capabilities.

      Step 1 — Installing Docker Machine

      In order to use Docker Machine, you must first install it locally. On Ubuntu, this means downloading a handful of scripts from the official Docker repository on GitHub.

      To download and install the Docker Machine binary, type:

      • wget https://github.com/docker/machine/releases/download/v0.15.0/docker-machine-$(uname -s)-$(uname -m)

      The name of the file should be docker-machine-Linux-x86_64. Rename it to docker-machine to make it easier to work with:

      • mv docker-machine-Linux-x86_64 docker-machine

      Make it executable:

      Move or copy it to the /usr/local/bin directory so that it will be available as a system command:

      • sudo mv docker-machine /usr/local/bin

      Check the version, which will indicate that it's properly installed:

      You'll see output similar to this, displaying the version number and build:

      Output

      docker-machine version 0.15.0, build b48dc28d

      Docker Machine is installed. Let's install some additional helper tools to make Docker Machine easier to work with.

      Step 2 — Installing Additional Docker Machine Scripts

      There are three Bash scripts in the Docker Machine GitHub repository you can install to make working with the docker and docker-machine commands easier. When installed, these scripts provide command completion and prompt customization.

      In this step, you'll install these three scripts into the /etc/bash_completion.d directory on your local machine by downloading them directly from the Docker Machine GitHub repository.

      Note: Before downloading and installing a script from the internet in a system-wide location, you should inspect the script's contents first by viewing the source URL in your browser.

      The first script allows you to see the active machine in your prompt. This comes in handy when you are working with and switching between multiple Dockerized machines. The script is called docker-machine-prompt.bash. Download it

      • sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine-prompt.bash -O /etc/bash_completion.d/docker-machine-prompt.bash

      To complete the installation of this file, you'll have to modify the value for the PS1 variable in your .bashrc file. The PS1 variable is a special shell variable used to modify the Bash command prompt. Open ~/.bashrc in your editor:

      Within that file, there are three lines that begin with PS1. They should look just like these:

      ~/.bashrc

      PS1='${debian_chroot:+($debian_chroot)}[33[01;32m]u@h[33[00m]:[33[01;34m]w[33[00m]$ '
      
      ...
      
      PS1='${debian_chroot:+($debian_chroot)}u@h:w$ '
      
      ...
      
      PS1="[e]0;${debian_chroot:+($debian_chroot)}u@h: wa]$PS1"
      

      For each line, insert $(__docker_machine_ps1 " [%s]") near the end, as shown in the following example:

      ~/.bashrc

      PS1='${debian_chroot:+($debian_chroot)}[33[01;32m]u@h[33[00m]:[33[01;34m]w[33[00m]$(__docker_machine_ps1 " [%s]")$ '
      
      ...
      
      PS1='${debian_chroot:+($debian_chroot)}u@h:w$(__docker_machine_ps1 " [%s]")$ '
      
      ...
      
      PS1="[e]0;${debian_chroot:+($debian_chroot)}u@h: wa]$(__docker_machine_ps1 " [%s]")$PS1"
      

      Save and close the file.

      The second script is called docker-machine-wrapper.bash. It adds a use subcommand to the docker-machine command, making it significantly easier to switch between Docker hosts. To download it, type:

      • sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine-wrapper.bash -O /etc/bash_completion.d/docker-machine-wrapper.bash

      The third script is called docker-machine.bash. It adds bash completion for docker-machine commands. Download it using:

      • sudo wget https://raw.githubusercontent.com/docker/machine/master/contrib/completion/bash/docker-machine.bash -O /etc/bash_completion.d/docker-machine.bash

      To apply the changes you've made so far, close, then reopen your terminal. If you're logged into the machine via SSH, exit the session and log in again, and you'll have command completion for the docker and docker-machine commands.

      Let's test things out by creating a new Docker host with Docker Machine.

      Step 3 — Provisioning a Dockerized Host Using Docker Machine

      Now that you have Docker and Docker Machine running on your local machine, you can provision a Dockerized Droplet on your DigitalOcean account using Docker Machine's docker-machine create command. If you've not done so already, assign your DigitalOcean API token to an environment variable:

      • export DOTOKEN=your-api-token

      NOTE: This tutorial uses DOTOKEN as the bash variable for the DO API token. The variable name does not have to be DOTOKEN, and it does not have to be in all caps.

      To make the variable permanent, put it in your ~/.bashrc file. This step is optional, but it is necessary if you want to the value to persist across shell sessions.

      Open that file with nano:

      Add this line to the file:

      ~/.bashrc

      export DOTOKEN=your-api-token
      

      To activate the variable in the current terminal session, type:

      To call the docker-machine create command successfully you must specify the driver you wish to use, as well as a machine name. The driver is the adapter for the infrastructure you're going to create. There are drivers for cloud infrastructure providers, as well as drivers for various virtualization platforms.

      We'll use the digitalocean driver. Depending on the driver you select, you'll need to provide additional options to create a machine. The digitalocean driver requires the API token (or the variable that evaluates to it) as its argument, along with the name for the machine you want to create.

      To create your first machine, type this command to create a DigitalOcean Droplet called docker-01:

      • docker-machine create --driver digitalocean --digitalocean-access-token $DOTOKEN docker-01

      You'll see this output as Docker Machine creates the Droplet:

      Output

      ... Installing Docker... Copying certs to the local machine directory... Copying certs to the remote machine... Setting Docker configuration on the remote daemon... Checking connection to Docker... Docker is up and running! To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env ubuntu1804-docker

      Docker Machine creates an SSH key pair for the new host so it can access the server remotely. The Droplet is provisioned with an operating system and Docker is installed. When the command is complete, your Docker Droplet is up and running.

      To see the newly-created machine from the command line, type:

      The output will be similar to this, indicating that the new Docker host is running:

      Output

      NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS docker-01 - digitalocean Running tcp://209.97.155.178:2376 v18.06.1-ce

      Now let's look at how to specify the operating system when we create a machine.

      Step 4 — Specifying the Base OS and Droplet Options When Creating a Dockerized Host

      By default, the base operating system used when creating a Dockerized host with Docker Machine is supposed to be the latest Ubuntu LTS. However, at the time of this publication, the docker-machine create command is still using Ubuntu 16.04 LTS as the base operating system, even though Ubuntu 18.04 is the latest LTS edition. So if you need to run Ubuntu 18.04 on a recently-provisioned machine, you'll have to specify Ubuntu along with the desired version by passing the --digitalocean-image flag to the docker-machine create command.

      For example, to create a machine using Ubuntu 18.04, type:

      • docker-machine create --driver digitalocean --digitalocean-image ubuntu-18-04-x64 --digitalocean-access-token $DOTOKEN docker-ubuntu-1804

      You're not limited to a version of Ubuntu. You can create a machine using any operating system supported on DigitalOcean. For example, to create a machine using Debian 8, type:

      • docker-machine create --driver digitalocean --digitalocean-image debian-8-x64 --digitalocean-access-token $DOTOKEN docker-debian

      To provision a Dockerized host using CentOS 7 as the base OS, specify centos-7-0-x86 as the image name, like so:

      • docker-machine create --driver digitalocean --digitalocean-image centos-7-0-x64 --digitalocean-access-token $DOTOKEN docker-centos7

      The base operating system is not the only choice you have. You can also specify the size of the Droplet. By default, it is the smallest Droplet, which has 1 GB of RAM, a single CPU, and a 25 GB SSD.

      Find the size of the Droplet you want to use by looking up the corresponding slug in the DigitalOcean API documentation.

      For example, to provision a machine with 2 GB of RAM, two CPUs, and a 60 GB SSD, use the slug s-2vcpu-2gb:

      • docker-machine create --driver digitalocean --digitalocean-size s-2vcpu-2gb --digitalocean-access-token $DOTOKEN docker-03

      To see all the flags specific to creating a Docker Machine using the DigitalOcean driver, type:

      • docker-machine create --driver digitalocean -h

      Tip: If you refresh the Droplet page of your DigitalOcean dashboard, you will see the new machines you created using the docker-machine command.

      Now let's explore some of the other Docker Machine commands.

      Step 5 — Executing Additional Docker Machine Commands

      You've seen how to provision a Dockerized host using the create subcommand, and how to list the hosts available to Docker Machine using the ls subcommand. In this step, you'll learn a few more useful subcommands.

      To obtain detailed information about a Dockerized host, use the inspect subcommand, like so:

      • docker-machine inspect docker-01

      The output includes lines like the ones in the following output. The Image line reveals the version of the Linux distribution used and the Size line indicates the size slug:

      Output

      ... { "ConfigVersion": 3, "Driver": { "IPAddress": "203.0.113.71", "MachineName": "docker-01", "SSHUser": "root", "SSHPort": 22, ... "Image": "ubuntu-16-04-x64", "Size": "s-1vcpu-1gb", ... }, ---

      To print the connection configuration for a host, type:

      • docker-machine config docker-01

      The output will be similar to this:

      Output

      --tlsverify --tlscacert="/home/kamit/.docker/machine/certs/ca.pem" --tlscert="/home/kamit/.docker/machine/certs/cert.pem" --tlskey="/home/kamit/.docker/machine/certs/key.pem" -H=tcp://203.0.113.71:2376

      The last line in the output of the docker-machine config command reveals the IP address of the host, but you can also get that piece of information by typing:

      • docker-machine ip docker-01

      If you need to power down a remote host, you can use docker-machine to stop it:

      • docker-machine stop docker-01

      Verify that it is stopped:

      The output shows that the status of the machine has changed:

      Ouput

      NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS docker-01 - digitalocean Stopped Unknown

      To start it again, use the start subcommand:

      • docker-machine start docker-01

      Then review its status again:

      You will see that the STATE is now set Running for the host:

      Ouput

      NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS docker-01 - digitalocean Running tcp://203.0.113.71:2376 v18.06.1-ce

      Next let's look at how to interact with the remote host using SSH.

      Step 6 — Executing Commands on a Dockerized Host via SSH

      At this point, you've been getting information about your machines, but you can do more than that. For example, you can execute native Linux commands on a Docker host by using the ssh subcommand of docker-machine from your local system. This section explains how to perform ssh commands via docker-machine as well as how to open an SSH session to a Dockerized host.

      Assuming that you've provisioned a machine with Ubuntu as the operating system, execute the following command from your local system to update the package database on the Docker host:

      • docker-machine ssh docker-01 apt-get update

      You can even apply available updates using:

      • docker-machine ssh docker-01 apt-get upgrade

      Not sure what kernel your remote Docker host is using? Type the following:

      • docker-machine ssh docker-01 uname -r

      Finally, you can log in to the remote host with the docker machine ssh command:

      docker-machine ssh docker-01
      

      You'll be logged in as the root user and you'll see something similar to the following:

      Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-131-generic x86_64)
      
       * Documentation:  https://help.ubuntu.com
       * Management:     https://landscape.canonical.com
       * Support:        https://ubuntu.com/advantage
      
        Get cloud support with Ubuntu Advantage Cloud Guest:
          http://www.ubuntu.com/business/services/cloud
      
      14 packages can be updated.
      10 updates are security updates.
      

      Log out by typing exit to return to your local machine.

      Next, we'll direct Docker's commands at our remote host.

      Step 7 — Activating a Dockerized Host

      Activating a Docker host connects your local Docker client to that system, which makes it possible to run normal docker commands on the remote system.

      First, use Docker Machine to create a new Docker host called docker-ubuntu using Ubuntu 18.04:

      • docker-machine create --driver digitalocean --digitalocean-image ubuntu-18-04-x64 --digitalocean-access-token $DOTOKEN docker-ubuntu

      To activate a Docker host, type the following command:

      • eval $(docker-machine env machine-name)

      Alternatively, you can activate it by using this command:

      • docker-machine use machine-name

      Tip When working with multiple Docker hosts, the docker-machine use command is the easiest method of switching from one to the other.

      After typing any of these commands, your prompt will change to indicate that your Docker client is pointing to the remote Docker host. It will take this form. The name of the host will be at the end of the prompt:

      username@localmachine:~ [docker-01]$
      

      Now any docker command you type at this command prompt will be executed on that remote host.

      Execute docker-machine ls again:

      You'll see an asterisk under the ACTIVE column for docker-01:

      Output

      NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS docker-01 * digitalocean Running tcp://203.0.113.71:2376 v18.06.1-ce

      To exit from the remote Docker host, type the following:

      Your prompt will no longer show the active host.

      Now let's create containers on the remote machine.

      Step 8 — Creating Docker Containers on a Remote Dockerized Host

      So far, you have provisioned a Dockerized Droplet on your DigitalOcean account and you've activated it — that is, your Docker client is pointing to it. The next logical step is to spin up containers on it. As an example, let's try running the official Nginx container.

      Use docker-machine use to select your remote machine:

      • docker-machine use docker-01

      Now execute this command to run an Nginx container on that machine:

      • docker run -d -p 8080:80 --name httpserver nginx

      In this command, we're mapping port 80 in the Nginx container to port 8080 on the Dockerized host so that we can access the default Nginx page from anywhere.

      Once the container builds, you will be able to access the default Nginx page by pointing your web browser to http://docker_machine_ip:8080.

      While the Docker host is still activated (as seen by its name in the prompt), you can list the images on that host:

      The output includes the Nginx image you just used:

      Output

      REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 71c43202b8ac 3 hours ago 109MB

      You can also list the active or running containers on the host:

      If the Nginx container you ran in this step is the only active container, the output will look like this:

      Output

      CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d3064c237372 nginx "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:8080->80/tcp httpserver

      If you intend to create containers on a remote machine, your Docker client must be pointing to it — that is, it must be the active machine in the terminal that you're using. Otherwise you'll be creating the container on your local machine. Again, let your command prompt be your guide.

      Docker Machine can create and manage remote hosts, and it can also remove them.

      Step 9 – Removing Docker Hosts

      You can use Docker Machine to remove a Docker host you've created. Use the docker-machine rm command to remove the docker-01 host you created:

      • docker-machine rm docker-01

      The Droplet is deleted along with the SSH key created for it. List the hosts again:

      This time, you won't see the docker-01 host listed in the output. And if you've only created one host, you won't see any output at all.

      Be sure to execute the command docker-machine use -u to point your local Docker daemon back to your local machine.

      Step 10 — Disabling Crash Reporting (Optional)

      By default, whenever an attempt to provision a Dockerized host using Docker Machine fails, or Docker Machine crashes, some diagnostic information is sent to a Docker account on Bugsnag. If you're not comfortable with this, you can disable the reporting by creating an empty file called no-error-report in your local computer's .docker/machine directory.

      To create the file, type:

      • touch ~/.docker/machine/no-error-report

      Check the file for error messages if provisioning fails or Docker Machine crashes.

      Conclusion

      You've installed Docker Machine and used it to provision multiple Docker hosts on DigitalOcean remotely from your local system. From here you should be able to provision as many Dockerized hosts on your DigitalOcean account as you need.

      For more on Docker Machine, visit the official documentation page. The three Bash scripts downloaded in this tutorial are hosted on this GitHub page.



      Source link

      How to Back Up and Restore a Kubernetes Cluster on DigitalOcean Using Heptio Ark


      Introduction

      Heptio Ark is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster’s Persistent Volumes using your cloud provider’s block storage snapshot features, and can then restore your cluster’s objects and Persistent Volumes to a previous state.

      StackPointCloud’s DigitalOcean Ark Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster’s state and restore it should disaster strike.

      In this tutorial we’ll set up and configure the Ark client on a local machine, and deploy the Ark server into our Kubernetes cluster. We’ll then deploy a sample Nginx app that uses a Persistent Volume for logging, and simulate a disaster recovery scenario.

      Prerequisites

      Before you begin this tutorial, you should have the following available to you:

      On your local computer:

      In your DigitalOcean account:

      • A DigitalOcean Kubernetes cluster, or a Kubernetes cluster (version 1.7.5 or later) on DigitalOcean Droplets
      • A DNS server running inside of your cluster. If you are using DigitalOcean Kubernetes, this is running by default. To learn more about configuring a Kubernetes DNS service, consult Customizing DNS Service from the official Kuberentes documentation.
      • A DigitalOcean Space that will store your backed-up Kubernetes objects. To learn how to create a Space, consult the Spaces product documentation.
      • An access key pair for your DigitalOcean Space. To learn how to create a set of access keys, consult How to Manage Administrative Access to Spaces.
      • A personal access token for use with the DigitalOcean API. To learn how to create a personal access token, consult How to Create a Personal Access Token.

      Once you have all of this set up, you’re ready to begin with this guide.

      Step 1 — Installing the Ark Client

      The Heptio Ark backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we’ll install the local Ark client.

      In your web browser, navigate to the Ark GitHub repo releases page, find the latest release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we’ll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine.

      Then, from the command line on your local computer, navigate to the temporary /tmp directory and cd into it:

      Use wget and the link you copied earlier to download the release tarball:

      • wget https://link_copied_from_release_page

      Once the download completes, extract the tarball using tar (note the filename may differ depending on the current release version and your OS):

      • tar -xvzf ark-v0.9.6-linux-amd64.tar.gz

      The /tmp directory should now contain the extracted ark binary as well as the tarball you just downloaded.

      Verify that you can run the ark client by executing the binary:

      You should see the following help output:

      Output

      Heptio Ark is a tool for managing disaster recovery, specifically for Kubernetes cluster resources. It provides a simple, configurable, and operationally robust way to back up your application state and associated data. If you're familiar with kubectl, Ark supports a similar model, allowing you to execute commands such as 'ark get backup' and 'ark create schedule'. The same operations can also be performed as 'ark backup get' and 'ark schedule create'. Usage: ark [command] Available Commands: backup Work with backups client Ark client related commands completion Output shell completion code for the specified shell (bash or zsh) create Create ark resources delete Delete ark resources describe Describe ark resources get Get ark resources help Help about any command plugin Work with plugins restic Work with restic restore Work with restores schedule Work with schedules server Run the ark server version Print the ark version and associated image . . .

      At this point you should move the ark executable out of the temporary /tmp directory and add it to your PATH. To add it to your PATH on an Ubuntu system, simply copy it to /usr/local/bin:

      • sudo mv ark /usr/local/bin/ark

      You're now ready to configure the Ark server and deploy it to your Kubernetes cluster.

      Step 2 — Installing and Configuring the Ark Server

      Before we deploy Ark into our Kubernetes cluster, we'll first create Ark's prerequisite objects. Ark's prerequisites consist of:

      • A heptio-ark Namespace

      • The ark Service Account

      • Role-based access control (RBAC) rules to grant permissions to the ark Service Account

      • Custom Resources (CRDs) for the Ark-specific resources: Backup, Schedule, Restore, Config

      A YAML file containing the specs for the above Kubernetes objects can be found in the official Ark Git repository. While still in the /tmp directory, download the Ark repo using git:

      • git clone https://github.com/heptio/ark.git

      Once downloaded, navigate into the ark directory:

      The prerequisite resources listed above can be found in the examples/common/00-prereqs.yaml YAML file. We'll create these resources in our Kubernetes cluster by using kubectl apply and passing in the file:

      • kubectl apply -f examples/common/00-prereqs.yaml

      You should see the following output:

      Output

      customresourcedefinition.apiextensions.k8s.io/backups.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/schedules.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/restores.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/configs.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/downloadrequests.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/deletebackuprequests.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/podvolumebackups.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/podvolumerestores.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/resticrepositories.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/backupstoragelocations.ark.heptio.com created namespace/heptio-ark created serviceaccount/ark created clusterrolebinding.rbac.authorization.k8s.io/ark created

      Now that we've created the necessary Ark Kubernetes objects in our cluster, we can download and install the Ark DigitalOcean Plugin, which will allow us to use DigitalOcean Spaces as a backupStorageProvider (for Kubernetes objects), and DigitalOcean Block Storage as a persistentVolumeProvider (for Persistent Volume backups).

      Move back out of the ark directory and fetch the plugin from StackPointCloud's repo using git:

      • cd ..
      • git clone https://github.com/StackPointCloud/ark-plugin-digitalocean.git

      Move into the plugin directory:

      • cd ark-plugin-digitalocean

      We'll now save the access keys for our DigitalOcean Space as a Kubernetes Secret. First, open up the examples/credentials-ark file using your favorite editor:

      • nano examples/credentials-ark

      Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your Spaces access key and secret key:

      examples/credentials-ark

      [default]
      aws_access_key_id=your_spaces_access_key_here
      aws_secret_access_key=your_spaces_secret_key_here
      

      Save and close the file.

      Now, create the cloud-credentials Secret using kubectl, inserting your Personal Access Token using the digitalocean_token data item:

      • kubectl create secret generic cloud-credentials
      • --namespace heptio-ark
      • --from-file cloud=examples/credentials-ark
      • --from-literal digitalocean_token=your_personal_access_token

      You should see the following output:

      Output

      secret/cloud-credentials created

      To confirm that the cloud-credentials Secret was created successfully, you can describe it using kubectl:

      • kubectl describe secrets/cloud-credentials --namespace heptio-ark

      You should see the following output describing the cloud-credentials secret:

      Output

      Name: cloud-credentials Namespace: heptio-ark Labels: <none> Annotations: <none> Type: Opaque Data ==== cloud: 115 bytes digitalocean_token: 64 bytes

      We can now move on to creating an Ark Config object named default. To do this, we'll edit a YAML configuration file and then create the object in our Kubernetes cluster.

      Open examples/10-ark-config.yaml in your favorite editor:

      • nano examples/10-ark-config.yaml

      Insert your Space's name and region in the highlighted fields:

      examples/10-ark-config.yaml

      ---
      apiVersion: ark.heptio.com/v1
      kind: Config
      metadata:
        namespace: heptio-ark
        name: default
      persistentVolumeProvider:
        name: digitalocean
      backupStorageProvider:
        name: aws
        bucket: space_name_here
        config:
          region: space_region_here
          s3ForcePathStyle: "true"
          s3Url: https://space_region_here.digitaloceanspaces.com
      backupSyncPeriod: 30m
      gcSyncPeriod: 30m
      scheduleSyncPeriod: 1m
      restoreOnlyMode: false
      

      persistentVolumeProvider sets DigitalOcean Block Storage as the the provider for Persistent Volume backups. These will be Block Storage Volume Snapshots.

      backupStorageProvider sets DigitalOcean Spaces as the provider for Kubernetes object backups. Ark will create a tarball of all your Kubernetes objects (or some, depending on how you execute it), and upload this tarball to Spaces.

      When you're done, save and close the file.

      Create the object in your cluster using kubectl apply:

      • kubectl apply -f examples/10-ark-config.yaml

      You should see the following output:

      Output

      config.ark.heptio.com/default created

      At this point, we've finished configuring the Ark server and can create its Kubernetes deployment, found in the examples/20-deployment.yaml configuration file. Let's take a quick look at this file:

      • cat examples/20-deployment.yaml

      You should see the following text:

      examples/20-deployment.yaml

      ---
      apiVersion: apps/v1beta1
      kind: Deployment
      metadata:
        namespace: heptio-ark
        name: ark
      spec:
        replicas: 1
        template:
          metadata:
            labels:
              component: ark
            annotations:
              prometheus.io/scrape: "true"
              prometheus.io/port: "8085"
              prometheus.io/path: "/metrics"
          spec:
            restartPolicy: Always
            serviceAccountName: ark
            containers:
              - name: ark
                image: gcr.io/heptio-images/ark:latest
                command:
                  - /ark
                args:
                  - server
                volumeMounts:
                  - name: cloud-credentials
                    mountPath: /credentials
                  - name: plugins
                    mountPath: /plugins
                  - name: scratch
                    mountPath: /scratch
                env:
                  - name: AWS_SHARED_CREDENTIALS_FILE
                    value: /credentials/cloud
                  - name: ARK_SCRATCH_DIR
                    value: /scratch
                  - name: DIGITALOCEAN_TOKEN
                    valueFrom:
                      secretKeyRef:
                        key: digitalocean_token
                        name: cloud-credentials
            volumes:
              - name: cloud-credentials
                secret:
                  secretName: cloud-credentials
              - name: plugins
                emptyDir: {}
              - name: scratch
                emptyDir: {}
      

      We observe here that we're creating a Deployment called ark that consists of a single replica of the gcr.io/heptio-images/ark:latest container. The Pod is configured using the cloud-credentials secret we created earlier.

      Create the Deployment using kubectl apply:

      • kubectl apply -f examples/20-deployment.yaml

      You should see the following output:

      Output

      deployment.apps/ark created

      We can double check that the Deployment has been successfully created using kubectl get on the heptio-ark Namespace :

      • kubectl get deployments --namespace=heptio-ark

      You should see the following output:

      Output

      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE ark 1 1 1 0 8m

      The Ark server Pod may not start correctly until you install the Ark DigitalOcean plugin. To install the ark-blockstore-digitalocean plugin, use the ark client we installed earlier:

      • ark plugin add quay.io/stackpoint/ark-blockstore-digitalocean:latest

      You can specify the kubeconfig to use with the --kubeconfig flag. If you don't use this flag, ark will check the KUBECONFIG environment variable and then fall back to the kubectl default (~/.kube/config).

      At this point Ark is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.

      In the next section, we'll run a quick test to make sure that the backup and restore functionality works as expected.

      Step 3 — Testing Backup and Restore Procedure

      Now that we've successfully installed and configured Ark, we can create a test Nginx Deployment and Persistent Volume, and run through a backup and restore drill to ensure that everything is working properly.

      The ark-plugin-digitalocean repository contains a sample Nginx deployment called nginx-pv.yaml.

      Let's take a quick look:

      • cat examples/nginx-pv.yaml

      You should see the following text:

      Output

      --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: nginx-logs namespace: nginx-example labels: app: nginx spec: storageClassName: do-block-storage accessModes: - ReadWriteOnce resources: requests: storage: 5Gi --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx-deployment namespace: nginx-example spec: replicas: 1 template: metadata: labels: app: nginx spec: volumes: - name: nginx-logs persistentVolumeClaim: claimName: nginx-logs containers: - image: nginx:1.7.9 name: nginx ports: - containerPort: 80 volumeMounts: - mountPath: "/var/log/nginx" name: nginx-logs readOnly: false --- apiVersion: v1 kind: Service metadata: labels: app: nginx name: my-nginx namespace: nginx-example spec: ports: - port: 80 targetPort: 80 selector: app: nginx type: LoadBalancer

      In this file, we observe specs for:

      • An Nginx Deployment consisting of a single replica of the nginx:1.7.9 container image
      • A 5Gi Persistent Volume Claim (called nginx-logs), using the do-block-storage StorageClass
      • A LoadBalancer Service that exposes port 80

      Create the deployment using kubectl apply:

      • kubectl apply -f examples/nginx-pv.yml

      You should see the following output:

      Output

      namespace/nginx-example created persistentvolumeclaim/nginx-logs created deployment.apps/nginx-deployment created service/my-nginx created

      Check that the Deployment succeeded:

      • kubectl get deployments --namespace=nginx-example

      You should see the following output:

      Output

      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 1 1 1 1 1h

      Once Available reaches 1, fetch the Nginx load balancer’s external IP using kubectl get:

      • kubectl get services --namespace=nginx-example

      You should see both the internal CLUSTER-IP and EXTERNAL-IP for the my-nginx Service:

      Output

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-nginx LoadBalancer 10.32.27.0 203.0.113.0 80:30754/TCP 3m

      Note the EXTERNAL-IP and navigate to it using your web browser.

      You should see the following NGINX welcome page:

      Nginx Welcome Page

      This indicates that your Nginx Deployment and Service are up and running.

      Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):

      Fetch the Pod’s name using kubectl get:

      • kubectl get pods --namespace nginx-example

      Output

      NAME READY STATUS RESTARTS AGE nginx-deployment-77d8f78fcb-zt4wr 1/1 Running 0 29m

      Now, exec into the running Nginx container to get a shell inside of it:

      • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

      Once inside the Nginx container, cat the Nginx access logs:

      • cat /var/log/nginx/access.log

      You should see some Nginx access entries:

      Output

      10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

      Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure.

      We can now perform the backup procedure to copy all nginx Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.

      We'll create a backup called nginx-backup using the ark client:

      • ark backup create nginx-backup --selector app=nginx

      The --selector app=nginx instructs the Ark server to only back up Kubernetes objects with the app=nginx Label Selector.

      You should see the following output:

      Output

      Backup request "nginx-backup" submitted successfully. Run `ark backup describe nginx-backup` for more details.

      Running ark backup describe nginx-backup should provide the following output after a short delay:

      Output

      Name: nginx-backup Namespace: heptio-ark Labels: <none> Annotations: <none> Phase: Completed Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: app=nginx Snapshot PVs: auto TTL: 720h0m0s Hooks: <none> Backup Format Version: 1 Started: 2018-09-26 00:14:30 -0400 EDT Completed: 2018-09-26 00:14:34 -0400 EDT Expiration: 2018-10-26 00:14:30 -0400 EDT Validation errors: <none> Persistent Volumes: pvc-e4862eac-c2d2-11e8-920b-92c754237aeb: Snapshot ID: 2eb66366-c2d3-11e8-963b-0a58ac14428b Type: ext4 Availability Zone: IOPS: <N/A>

      This output indicates that nginx-backup completed successfully.

      From the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.

      You should see a new directory called nginx-backup containing the Ark backup files.

      Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.

      We can now test the restore procedure.

      Let's first delete the nginx-example Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:

      • kubectl delete namespace nginx-example

      Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example Deployment is no longer running:

      • kubectl get deployments --namespace=nginx-example

      Output

      No resources found.

      We can now perform the restore procedure, once again using the ark client:

      • ark restore create --from-backup nginx-backup

      Here we use create to create an Ark Restore object from the nginx-backup object.

      You should see the following output:

      Output

      • Restore request "nginx-backup-20180926143828" submitted successfully.
      • Run `ark restore describe nginx-backup-20180926143828` for more details.

      Check the status of the restored Deployment:

      • kubectl get deployments --namespace=nginx-example

      Output

      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 1 1 1 1 1m

      Check for the creation of a Persistent Volume:

      • kubectl get pvc --namespace=nginx-example

      Output

      NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nginx-logs Bound pvc-e4862eac-c2d2-11e8-920b-92c754237aeb 5Gi RWO do-block-storage 3m

      Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.

      Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.

      To do this, once again fetch the Pod’s name using kubectl get:

      • kubectl get pods --namespace nginx-example

      Output

      NAME READY STATUS RESTARTS AGE nginx-deployment-77d8f78fcb-zt4wr 1/1 Running 0 29m

      Then exec into it:

      • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

      Once inside the Nginx container, cat the Nginx access logs:

      • cat /var/log/nginx/access.log

      Output

      10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

      You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.

      At this point, we've successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.

      Conclusion

      In this guide we installed and configured the Ark Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.

      Ark can also be used to schedule regular backups of your Kubernetes cluster. To do this, you can use the ark schedule command. It can also be used to migrate resources from one cluster to another. To learn more about these two use cases, consult the official Ark documentation.

      To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.

      This tutorial builds on the README found in StackPointCloud's ark-plugin-digitalocean GitHub repo.



      Source link