One place for hosting & domains


      How to Use Buildah to Build OCI Container Images

      Buildah is an open source containerization tool capable of creating images from scratch, Dockerfiles, or Containerfiles. It also follows the Open Container Initiative (OCI) specifications, making Buildah images both versatile and open.

      Learn how to install and start using Buildah in this tutorial. Below, find steps for creating containers and rendering those containers to images.

      Before You Begin

      1. Familiarize yourself with our
        Getting Started with Linode guide, and complete the steps for setting your Linode’s hostname and timezone.

      2. This guide uses sudo wherever possible. Complete the sections of our
        How to Secure Your Server guide to create a standard user account, harden SSH access, and remove unnecessary network services.

      3. Update your system.

        • AlmaLinux, CentOS Stream, Fedora, or Rocky Linux:

          sudo dnf upgrade
        • Ubuntu:

          sudo apt update && sudo apt upgrade


      This guide is written for a non-root user. Commands that require elevated privileges are prefixed with sudo. If you’re not familiar with the sudo command, see the
      Users and Groups guide.

      What Is Buildah?

      Buildah is an open source tool for building container images that are compliant with the OCI.

      The OCI seeks to create an open standard for containerization. To that end, it defines specifications for container runtimes and images. Another goal of the OCI is to help secure and streamline operating system virtualization.

      Buildah provides powerful tools to create and maintain OCI-compliant images. You may be familiar with Dockerfiles, one of the most common formats for container images. Buildah fully supports them, and can create images directly from them.

      But Buildah can also craft container images from scratch. Buildah allows you to use the command line to build up the container from a complete blank slate, giving it only the contents you need. Buildah can then render and export an OCI container image from your work.

      Buildah vs Docker

      Overall, Buildah is similar in functionality to Docker. So what sets it apart? Why use Buildah instead of Docker?

      One of Buildah’s primary advantages is it avoids the security risks of the Docker daemon. The Docker daemon runs on a socket with root-level access, and this has the potential to introduce security risks. Buildah avoids this risk by running without a daemon, allowing containers to be truly rootless.

      With Buildah, the user also has the ability to create container images from scratch. Buildah can mount an empty container and let the user add only what they need. This feature can be extraordinarily useful when you need a lightweight image.

      Buildah also gives the user precise control of images, and specifically image layers. For those wanting more capabilities in their containerization tools, Buildah tends to offer what they need.

      However, Buildah is not as useful when it comes to running and deploying container images. It can run them, but lacks some of the features to be found in other tools. Instead, Buildah puts the vast majority of its emphasis on creating containers and building container images.

      For that reason, users often build their OCI images in Buildah and run them using Podman, a tool for running and managing containers. You can learn more about Podman in our guide
      Podman vs Docker: Comparing the Two Containerization Tools.

      How to Install Buildah

      1. Install Buildah using your distribution’s package manager.

        • AlmaLinux, CentOS Stream (8 or later), Fedora, or Rocky Linux:

          sudo dnf install buildah
        • Ubuntu (20.10 or later):

          sudo apt install buildah
      2. Verify your installation by checking the installed Buildah version using the command below:

        buildah -v

        Your output may vary from what is shown here, but you are just looking to see that Buildah installed successfully:

        buildah version 1.26.1 (image-spec 1.0.2-dev, runtime-spec 1.0.2-dev)

      Configuring Buildah for Rootless Usage

      By default, Buildah commands are executed with root privileges, prefaced with the sudo command. However, one of the most appealing features of Buildah is its ability to run containers in rootless mode. This lets limited users work securely with Buildah.

      While Docker also allows you to run commands as a limited user, the Docker daemon still runs as root. This is a potential security issue with Docker, one that may allow limited users to execute privileged commands through the daemon.

      Buildah’s rootless mode solves this because it runs containers completely in a non-root environment, without a root daemon. Find the steps needed to set up your Buildah instance for rootless usage below.

      1. Install the slirp4netns and fuse-overlayfs tools to support your rootless Buildah operations.

        • AlmaLinux, CentOS Stream, Fedora, or Rocky Linux:

          sudo dnf install slirp4netns fuse-overlayfs
        • Ubuntu:

          sudo apt install slirp4netns fuse-overlayfs
      2. Add subuids and subgids ranges for your limited user. This example does so for the user example_user. It gives that user a sub-UID and sub-GID of 100000, each with a range of 65535 IDs:

        sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 example_user

      How to Use Buildah

      Buildah is primarily used for creating container images. Like Docker, Buildah can construct containers from Dockerfiles, but Buildah stands out for also allowing you to craft images from scratch.

      The next two sections show you how to build container images using each of these methods.

      Creating an Image from a Dockerfile

      Dockerfiles provide an approachable way to create containers with Buildah, especially for users already familiar with Docker or Dockerfiles.

      Buildah is fully capable of interpreting Dockerfile script, making it straightforward to build your Docker container images with Buildah.

      This guide uses an example Dockerfile provided in one of the official Buildah tutorials. This Dockerfile results in a container with the latest version of Fedora and the Apache HTTP server (httpd). It also “exposes” the HTTP server via port 80.

      1. Create a new file named Dockerfile in your user’s home directory:

        nano Dockerfile
      2. Fill it with the following contents:

        File: Dockerfile
        # Base on the most recently released Fedora
        FROM fedora:latest
        MAINTAINER ipbabble email [email protected] # not a real email
        # Install updates and httpd
        RUN echo "Updating all fedora packages"; dnf -y update; dnf -y clean all
        RUN echo "Installing httpd"; dnf -y install httpd && dnf -y clean all
        # Expose the default httpd port 80
        EXPOSE 80
        # Run the httpd
        CMD ["/usr/sbin/httpd", "-DFOREGROUND"]
      3. Press CTRL+X to exit, Y to save, and Enter to quit nano.

        Assuming you are still in the directory where this Dockerfile is located (your user’s home directory), you can immediately build the container’s image.

      4. This example names the new image fedora-http-server:

        buildah build -t fedora-http-server

        The output should look like the following:

        STEP 1/6: FROM fedora:latest
        Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
        Trying to pull
        Getting image source signatures
        Copying blob 75f075168a24 done
        Copying config 3a66698e60 done
        Writing manifest to image destination
        Storing signatures
        STEP 2/6: MAINTAINER ipbabble email [email protected] # not a real email
        STEP 3/6: RUN echo "Updating all fedora packages"; dnf -y update; dnf -y clean all

        Now you can now run the image with Podman, a tool for working with containers which is often used as a compliment to Buildah.

      5. First, install Podman:

        • AlmaLinux, CentOS Stream, Fedora, or Rocky Linux:

          sudo dnf install podman
        • Ubuntu:

          sudo apt install podman
      6. In the command below, the -p option “publishes” a given port, here routing the container’s port 80 to the local machine’s port 8080. The --rm option automatically removes the container when it has finished running, a fitting solution for a quick test like this.

        podman run -p 8080:80 --rm fedora-http-server
      7. Now you can open another Terminal session on the machine where the image is running, and use a cURL command to verify the default page is being served on port 8080:

        curl localhost:8080

        You should see the raw HTML of the Fedora HTTP Server test page as output:

        <!doctype html>
            <meta charset='utf-8'>
            <meta name='viewport' content='width=device-width, initial-scale=1'>
            <title>Test Page for the HTTP Server on Fedora</title>
            <style type="text/css">
              html {
                height: 100%;
                width: 100%;
                body {
      8. When done, stop the container, but first, determine your container’s ID or name:

        podman ps

        You should see an out put like this:

        CONTAINER ID  IMAGE                                COMMAND               CREATED        STATUS            PORTS                 NAMES
        daadb647b880  localhost/fedora-http-server:latest  /usr/sbin/httpd -...  8 seconds ago  Up 8 seconds ago>80/tcp  suspicious_goodall
      9. Now stop the container. Replace container-name-or-id with your container name or ID:

        podman stop container-name-or-id

        Since we set this example container to automatically remove when done with the --rm flag, stopping it also removes it.

      10. You can now logout, close the second Terminal session, and return to the original Terminal:


      Learn more about Podman in our guide
      How to Install Podman for Running Containers.

      You can also learn more about crafting Dockerfiles in our guide
      How to Use a Dockerfile to Build a Docker Image. This guide also includes links to further tutorials with more in-depth coverage of Dockerfiles.

      Creating an Image from Scratch

      As noted above, Buildah stands out for its ability to create container images from scratch. This section walks you through an example of how you can do just that.


      Buildah’s commands for working with containers can involve a few keywords, so often these commands are executed using environment variables. So, for instance, to create a new container with Fedora, you may see something like:

      fedoracontainer=$(buildah from fedora)

      Learn more about how environment variables work in our guide
      How to Use and Set Environment Variables.

      The example container that follows starts with an empty container. It then adds Bash and some other core utilities to that container to demonstrate how you can add programs to create a minimal container image.


      This section assumes you want to run Buildah in rootless mode, being its major draw versus Docker. Unfortunately, the Ubuntu package manager, APT, presents issues with installing packages onto a non-root container. So the instructions that follow are for RHEL-derived distributions such as AlmaLinux, CentOS Stream, Fedora, and Rocky Linux.

      If you want to run Buildah under Ubuntu in regular root mode, simply preface each buildah command that follows with sudo.

      For rootless operation, you need to execute the unshare command first. This command puts you in a shell within the user namespace. The next several steps presume your are in the user namespace shell until noted, otherwise the buildah mount command below will fail.

      1. Enter the user namespace shell:

        buildah unshare
      2. Create a blank container using Buildah’s scratch base:

        scratchcontainer=$(buildah from scratch)
      3. Mount the container as a virtual file system:

        scratchmnt=$(buildah mount $scratchcontainer)
      4. Install Bash and coreutils to the empty container.

        • AlmaLinux, CentOS Stream, Fedora, or Rocky Linux:

          Replace the value 36 below with the version of your RHEL-derived distribution:

          dnf install --installroot $scratchmnt --releasever 36 bash coreutils --setopt install_weak_deps=false
        • Debian or Ubuntu:

          Replace the value bullseye below with the codename of your Debian-based distribution:

          sudo apt install debootstrap
          sudo debootstrap bullseye $scratchmnt
      5. You can now test Bash on the container. The following command puts you in a Bash shell within the container:

        buildah run $scratchcontainer bash
      6. You can then exit the Bash shell using:

      7. You can now safely operate the container from outside of the user namespace shell initiated with unshare:


        From here on out, we replace $scratchcontainer with the container’s name, which should be working-container. However, if you have more than one container, the container’s name may differ. You can verify the container name via the buildah containers command.

      8. Now let’s recreate the test script file. From your user’s home directory, create the script-files folder and the file in the script-files folder:

        mkdir script-files
        nano script-files/

        Give it the following contents:

        File: script-files/
        echo "This is an example script."

        When done, press CTRL+X to exit, Y to save, and Enter to quit.

      9. The command below copies that file to the container’s /usr/bin directory:

        buildah copy working-container ~/script-files/ /usr/bin
      10. Verify the file’s delivery by running the ls command on the container for the /usr/bin directory:

        buildah run working-container ls /usr/bin

        Your file should be among the listed files:

      11. For a working example of how to execute scripts on a Buildah container, give this file executable permissions:

        buildah run working-container chmod +x /usr/bin/
      12. You can now run the script via the run command:

        buildah run working-container /usr/bin/

        Your output should be identical to the following:

        This is an example script.
      13. Once you are satisfied with the container, you can commit the change to an image:

        buildah commit working-container bash-core-image

        Your output should look something like this:

        Getting image source signatures
        Copying blob a0282af9505e done
        Copying config 9ea7958840 done
        Writing manifest to image destination
        Storing signatures
      14. You can now unmount and remove the container:

        buildah unmount working-container
        buildah rm working-container

      Managing Images and Containers

      Buildah is oriented towards creating container images, but it does have a few features for reviewing available containers and images. Here’s a brief list of the associated commands for these features.

      • To see a list of images built with your Buildah instance, run the following command:

        buildah images

        If you followed along for the sections above on creating Buildah images, you may have an image listing like this:

        REPOSITORY                  TAG      IMAGE ID       CREATED              SIZE
        localhost/fedora-http-server        latest   c313b363840d   8 minutes ago    314 MB
        localhost/bash-core-image           latest   9ea79588405b   20 minutes ago   108 MB   latest   3a66698e6040   2 months ago     169 MB
      • To list containers currently running under Buildah, use the following command:

        buildah containers

        Should you use this command while the container is still running from the section above on building an image from scratch, you may get an output like:

        68a1cc02025d     *                  scratch                          working-container
      • You can get the details of a particular image using a command like the following one, replacing 9ea79588405b with your image’s ID. You can get your image’s ID when the image is built or from the buildah images command show above:

        buildah inspect 9ea79588405b

        The image details actually consist of the JSON document that fully represents the image’s contents. All container images are just that: JSON documents with the instructions for building their corresponding containers.

        Here is an example of the first portion of a container image JSON resulting from the section above on creating an image from scratch:

            "Type": "buildah 0.0.1",
            "FromImage": "localhost/bash-core-image:latest",
            "FromImageID": "9ea79588405b48ff7b0572438a81a888c2eb25d95e6526b75b1020108ac11c10",
            "FromImageDigest": "sha256:beee0e0603e62647addab15341f1a52361a9684934d8d6ecbe1571fabd083dca",
            "Config": "{\"created\":\"2022-07-20T17:34:55.16639723Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"Labels\":{\"io.buildah.version\":\"1.26.1\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:a0282af9505ed0545c7fb82e1408e1b130cad13a9c3393870c7c4a0d5cf06a62\"]},\"history\":[{\"created\":\"2022-07-20T17:34:55.72288433Z\",\"created_by\":\"/bin/sh\"}]}",
            "Manifest": "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.oci.image.manifest.v1+json\",\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:9ea79588405b48ff7b0572438a81a888c2eb25d95e6526b75b1020108ac11c10\",\"size\":324},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:a0282af9505ed0545c7fb82e1408e1b130cad13a9c3393870c7c4a0d5cf06a62\",\"size\":108421632}],\"annotations\":{\"org.opencontainers.image.base.digest\":\"\",\"\":\"\"}}",
            "Container": "",
            "ContainerID": "",
            "MountPoint": "",
            "ProcessLabel": "",
            "MountLabel": "",
            "ImageAnnotations": {
                "org.opencontainers.image.base.digest": "",
                "": ""


      Buildah gives you a simple yet robust tool for crafting container images. It’s more than just an alternative to Docker. Buildah is a containerization tool for securely creating open containers and container images. With this tutorial, you have everything you need to get started building your own images and using Buildah to the utmost.

      More Information

      You may wish to consult the following resources for additional information
      on this topic. While these are provided in the hope that they will be
      useful, please note that we cannot vouch for the accuracy or timeliness of
      externally hosted materials.

      Source link

      Supercharging How You Build Apps With Little to No Code

      How to Join

      This Tech Talk is free and open to everyone. Register below to get a link to join the live stream or receive the video recording after it airs.

      February 16, 202111:00–12:00 p.m. ET / 4:00–5:00 p.m. GMT

      About the Talk

      Wish it were easier to tinker with code, collaborate, and scale?

      Explore how you can unlock more magic and creativity with Glitch and DigitalOcean.

      Join Anil (CEO, Glitch) and Gabe (CPO, DigitalOcean) in a conversation about simplifying your workflow while you get apps to market faster.

      What You’ll Learn

      • Experimenting without dealing with the headaches of coding
      • Going from idea to code, with the support of a community of builders
      • Simplifying your development workflow and cloud setup so that you can build fast, full-stack apps in the browser, and scale with confidence

      This Talk is Designed For

      Anyone who wants to build fast, full-stack web apps in the browser while keeping costs in check


      Getting Started on Glitch
      Glitch is a friendly community that makes it super easy to build a full-stack app and launch it on a secure URL in under a minute. Under the hood, Glitch’s in-browser editor combines real-time code collaboration, instant deployment, and automatic git-based version control. On the surface it makes all of that as easy as pushing a button.

      The DigitalOcean Developer Cloud™ simplifies modern app creation for new generations of developers working independently, at tech startups, and within small-to-medium sized businesses around the world.

      About the Presenters

      Anil Dash is CEO of Glitch, the friendly developer community where coders have collaborated to create and share millions of web apps. He is recognized as a leading advocate for more humane, inclusive and ethical technology through his work as an entrepreneur, activist and writer.

      Gabe Monroy is Chief Product Officer at DigitalOcean where he oversees the product management organization and owns the company’s product vision. Previously, Gabe was vice president of the Azure Developer Experience group at Microsoft, and founder and chief technology officer (CTO) of Deis, which was acquired by Microsoft in 2017. After joining Microsoft, Gabe started Azure Kubernetes Service, the fastest-growing compute service in Azure’s history. As an early contributor to Docker and Kubernetes, and a founding member of the Cloud Native Computing Foundation (CNCF), Gabe has deep experience in open source software and cloud-native technology.

      Need help with large deployments, migration planning, or a proof of concept? Fill out this form and we’ll be in touch!

      Source link

      How To Build A Security Information and Event Management (SIEM) System with Suricata and the Elastic Stack on Rocky Linux 8

      Not using Rocky Linux 8?

      Choose a different version or distribution.


      The previous tutorials in this series guided you through installing, configuring, and running Suricata as an Intrusion Detection (IDS) and Intrusion Prevention (IPS) system. You also learned about Suricata rules and how to create your own.

      In this tutorial you will explore how to integrate Suricata with Elasticsearch, Kibana, and Filebeat to begin creating your own Security Information and Event Management (SIEM) tool using the Elastic stack and Rocky Linux 8. SIEM tools are used to collect, aggregate, store, and analyze event data to search for security threats and suspicious activity on your networks and servers.

      The components that you will use to build your own SIEM are:

      • Elasticsearch to store, index, correlate, and search the security events that come from your Suricata server.
      • Kibana to display and navigate around the security event logs that are stored in Elasticsearch.
      • Filebeat to parse Suricata’s eve.json log file and send each event to Elasticsearch for processing.
      • Suricata to scan your network traffic for suspicious events, and either log or drop invalid packets.

      First you’ll install and configure Elasticsearch and Kibana with some specific authentication settings. Then you’ll add Filebeat to your Suricata system to send its eve.json logs to Elasticsearch.

      Finally, you’ll learn how to connect to Kibana using SSH and your web browser, and then load and interact with Kibana dashboards that show Suricata’s events and alerts.


      If you have been following this tutorial series then you should already have Suricata running on a Rocky Linux server. This server will be referred to as your Suricata server.

      You will also need a second server to host Elasticsearch and Kibana. This server will be referred to as your Elasticsearch server. It should be a Rocky Linux 8 server with:

      For the purposes of this tutorial, both servers should be able to communicate using private IP addresses. You can use a VPN like WireGuard to connect your servers, or use a cloud-provider that has private networking between hosts. You can also choose to run Elasticsearch, Kibana, Filebeat, and Suricata on the same server for experimenting.

      Step 1 — Installing Elasticsearch and Kibana

      The first step in this tutorial is to install Elasticsearch and Kibana on your Elasticsearch server. To get started, add the Elastic GPG key to your server with the following command:

      • sudo rpm --import

      Next, create an elasticsearch.repo file in your /etc/yum/yum.repos.d directory with the following contents, using vi or your preferred editor. This ensures that the upstream Elasticsearch repositories will be used when installing new packages via yum:

      • sudo vi /etc/yum.repos.d/elasticsearch.repo


      name=Elasticsearch repository for 7.x packages

      If you are using vi, when you are finished making changes, press ESC and then :x to write the changes to the file and quit.

      Now install Elasticsearch and Kibana using the dnf command. Press Y to accept any prompts about GPG key fingerprints:

      • sudo dnf install --enablerepo=elasticsearch elasticsearch kibana

      The --enablerepo option is used to override the default disabled setting in the /etc/yum.repos.d/elasticsearch.repo file. This approach ensures that the Elasticsearch and Kibana packages do not get accidentally upgraded when you install other package updates to your server.

      Once you are done installing the packages, find and record your server’s private IP address using the ip address show command:

      You will receive output like the following:


      lo UNKNOWN ::1/128 eth0 UP 2604:a880:cad:d0::e56:8001/64 fe80::b832:69ff:fe46:7e5d/64 eth1 UP fe80::b883:5bff:fe19:43f3/64

      The private network interface in this output is the highlighted eth1 device, with the IPv4 address Your device name, and IP addresses will be different. Regardless of your device name and private IP address, the address will be from the following reserved blocks:

      • to (10/8 prefix)
      • to (172.16/12 prefix)
      • to (192.168/16 prefix)

      If you would like to learn more about how these blocks are allocated visit the RFC 1918 specification)

      Record the private IP address for your Elasticsearch server (in this case This address will be referred to as your_private_ip in the remainder of this tutorial. Also note the name of the network interface, in this case eth1. In the next part of this tutorial you will configure Elasticsearch and Kibana to listen for connections on the private IP address coming from your Suricata server.

      Step 2 — Configuring Elasticsearch

      Elasticsearch is configured to only accept local connections by default. Additionally, it does not have any authentication enabled, so tools like Filebeat will not be able to send logs to it. In this section of the tutorial you will configure the network settings for Elasticsearch and then enable Elasticsearch’s built-in xpack security module.

      Configuring Elasticsearch Networking

      Since Your Elasticsearch and Suricata servers are separate, you will need to configure Elasticsearch to listen for connections on its private network interface.

      Open the /etc/elasticsearch/elasticsearch.yml file using vi or your preferred editor:

      • sudo vi /etc/elasticsearch/elasticsearch.yml

      Find the commented out line between lines 50–60 and add a new line after it that configures the network.bind_host setting, as highlighted below:

      # By default Elasticsearch is only accessible on localhost. Set a different
      # address here to expose this node on the network:
      network.bind_host: ["", "your_private_ip"]
      # By default Elasticsearch listens for HTTP traffic on the first free port it
      # finds starting at 9200. Set a specific HTTP port here:

      Substitute your private IP in place of the your_private_ip address. This line will ensure that Elasticsearch is still available on its local address so that Kibana can reach it, as well as on the private IP address for your server.

      Next, go to the end of the file using the vi shortcut SHIFT+G.

      Add the following highlighted lines to the end of the file:

      . . .
      discovery.type: single-node true

      The discovery.type setting allows Elasticsearch to run as a single node, as opposed to in a cluster of other Elasticsearch servers. The setting turns on some of the security features that are included with Elasticsearch.

      Save and close the file when you are done editing it.

      Finally, add firewall rules to ensure your Elasticsearch server is reachable on its private network interface. If you followed the prerequisite tutorials and are using firewalld, run the following commands:

      • sudo firewall-cmd --permanent --zone=internal --change-interface=eth1
      • sudo firewall-cmd --permanent --zone=internal --add-service=elasticsearch
      • sudo firewall-cmd --permanent --zone=internal --add-service=kibana
      • sudo systemctl reload firewalld.service

      Substitute your private network interface name in place of eth1 in the first command if yours is different. That command changes the interface rules to use the internal Firewalld zone, which is more permissive than the default public zone.

      The next commands add rules to allow Elasticsearch traffic on port 9200 and 9300, along with Kibana traffic on port 5601.

      The final command reloads the Firewalld service with the new permanent rules in place.

      Next you will start the Elasticsearch daemon and then configure passwords for use with the xpack security module.

      Starting Elasticsearch

      Now that you have configured networking and the xpack security settings for Elasticsearch, you need to start it for the changes to take effect.

      Run the following systemctl command to start Elasticsearch:

      • sudo systemctl start elasticsearch.service

      Once Elasticsearch finishes starting, you can continue to the next section of this tutorial where you will generate passwords for the default users that are built-in to Elasticsearch.

      Configuring Elasticsearch Passwords

      Now that you have enabled the setting, you need to generate passwords for the default Elasticsearch users. Elasticsearch includes a utility in the /usr/share/elasticsearch/bin directory that can automatically generate random passwords for these users.

      Run the following command to cd to the directory and then generate random passwords for all the default users:

      • cd /usr/share/elasticsearch/bin
      • sudo ./elasticsearch-setup-passwords auto

      You will receive output like the following. When prompted to continue, press y and then RETURN or ENTER:

      Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
      The passwords will be randomly generated and printed to the console.
      Please confirm that you would like to continue [y/N]y
      Changed password for user apm_system
      PASSWORD apm_system = eWqzd0asAmxZ0gcJpOvn
      Changed password for user kibana_system
      PASSWORD kibana_system = 1HLVxfqZMd7aFQS6Uabl
      Changed password for user kibana
      PASSWORD kibana = 1HLVxfqZMd7aFQS6Uabl
      Changed password for user logstash_system
      PASSWORD logstash_system = wUjY59H91WGvGaN8uFLc
      Changed password for user beats_system
      PASSWORD beats_system = 2p81hIdAzWKknhzA992m
      Changed password for user remote_monitoring_user
      PASSWORD remote_monitoring_user = 85HF85Fl6cPslJlA8wPG
      Changed password for user elastic
      PASSWORD elastic = 6kNbsxQGYZ2EQJiqJpgl

      You will not be able to run the utility again, so make sure to record these passwords somewhere secure. You will need to use the kibana_system user’s password in the next section of this tutorial, and the elastic user’s password in the Configuring Filebeat step of this tutorial.

      At this point in the tutorial you are finished configuring Elasticsearch. The next section explains how to configure Kibana’s network settings and its xpack security module.

      Step 3 — Configuring Kibana

      In the previous section of this tutorial, you configured Elasticsearch to listen for connections on your Elasticsearch server’s private IP address. You will need to do the same for Kibana so that Filebeats on your Suricata server can reach it.

      First you’ll enable Kibana’s xpack security functionality by generating some secrets that Kibana will use to store data in Elasticsearch. Then you’ll configure Kibana’s network setting and authentication details to connect to Elasticsearch.

      Enabling in Kibana

      To get started with xpack security settings in Kibana, you need to generate some encryption keys. Kibana uses these keys to store session data (like cookies), as well as various saved dashboards and views of data in Elasticsearch.

      You can generate the required encryption keys using the kibana-encryption-keys utility that is included in the /usr/share/kibana/bin directory. Run the following to cd to the directory and then generate the keys:

      • cd /usr/share/kibana/bin/
      • sudo ./kibana-encryption-keys generate -q --force

      The -q flag suppresses the tool’s instructions, and the --force flag will ensure that you create new keys. You should receive output like the following:


      xpack.encryptedSavedObjects.encryptionKey: 66fbd85ceb3cba51c0e939fb2526f585 xpack.reporting.encryptionKey: 9358f4bc7189ae0ade1b8deeec7f38ef 8f847a594e4a813c4187fa93c884e92b

      Copy these three keys somewhere secure. You will now add them to Kibana’s /etc/kibana/kibana.yml configuration file.

      Open the file using vi or your preferred editor:

      • sudo vi /etc/kibana/kibana.yml

      Go to the end of the file using the vi shortcut SHIFT+G. Paste the three xpack lines that you copied to the end of the file:


      . . .
      # Specifies locale to be used for all localizable strings, dates and number formats.
      # Supported languages are the following: English - en , by default , Chinese - zh-CN .
      #i18n.locale: "en"
      xpack.encryptedSavedObjects.encryptionKey: 66fbd85ceb3cba51c0e939fb2526f585
      xpack.reporting.encryptionKey: 9358f4bc7189ae0ade1b8deeec7f38ef 8f847a594e4a813c4187fa93c884e92b

      Keep the file open and proceed to the next section where you will configure Kibana’s network settings.

      Configuring Kibana Networking

      To configure Kibana’s networking so that it is available on your Elasticsearch server’s private IP address, find the commented out "localhost" line in /etc/kibana/kibana.yml. The line is near the beginning of the file. Add a new line after it with your server’s private IP address, as highlighted below:


      # Kibana is served by a back end server. This setting specifies the port to use.
      #server.port: 5601
      # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
      # The default is 'localhost', which usually means remote machines will not be able to connect.
      # To allow connections from remote users, set this parameter to a non-loopback address. "localhost" "your_private_ip"

      Substitute your private IP in place of the your_private_ip address.

      Save and close the file when you are done editing it. Next, you’ll need to configure the username and password that Kibana uses to connect to Elasticsearch.

      Configuring Kibana Credentials

      There are two ways to set the username and password that Kibana uses to authenticate to Elasticsearch. The first is to edit the /etc/kibana/kibana.yml configuration file and add the values there. The second method is to store the values in Kibana’s keystore, which is an obfuscated file that Kibana can use to store secrets.

      We’ll use the keystore method in this tutorial since it avoids editing Kibana’s configuration file directly.

      If you prefer to edit the file instead, the settings to configure in it are elasticsearch.username and elasticsearch.password.

      If you choose to edit the configuration file, skip the rest of the steps in this section.

      To add a secret to the keystore using the kibana-keystore utility, first cd to the the /usr/share/kibana/bin directory. Next, run the following command to set the username for Kibana:

      • cd /usr/share/kibana/bin
      • sudo ./kibana-keystore add elasticsearch.username

      You will receive a prompt like the following:

      Username Entry

      Enter value for elasticsearch.username: *************

      Enter kibana_system when prompted, either by copying and pasting, or typing the username carefully. Each character that you type will be masked with an * asterisk character. Press ENTER or RETURN when you are done entering the username.

      Now repeat the process, this time to save the password. Be sure to copy the password for the kibana_system user that you generated in the previous section of this tutorial. For reference, in this tutorial the example password is 1HLVxfqZMd7aFQS6Uabl.

      Run the following command to set the password:

      • sudo ./kibana-keystore add elasticsearch.password

      When prompted, paste the password to avoid any transcription errors:

      Password Entry

      Enter value for elasticsearch.password: ********************

      Starting Kibana

      Now that you have configured networking and the xpack security settings for Kibana, as well as added credentials to the keystore, you need to start it for the changes to take effect.

      Run the following systemctl command to restart Kibana:

      • sudo systemctl start kibana.service

      Once Kibana starts, you can continue to the next section of this tutorial where you will configure Filebeat on your Suricata server to send its logs to Elasticsearch.

      Step 4 — Installing Filebeat

      Now that your Elasticsearch and Kibana processes are configured with the correct network and authentication settings, the next step is to install and set up Filebeat on your Suricata server.

      To get started installing Filebeat, add the Elastic GPG key to your Suricata server with the following command:

      • sudo rpm --import

      Next, create an elasticsearch.repo file in your /etc/yum/yum.repos.d directory with the following contents, using vi or your preferred editor:

      • sudo vi /etc/yum.repos.d/elasticsearch.repo


      name=Elasticsearch repository for 7.x packages

      When you are finished making changes save and exit the file. Now install the Filebeat package using the dnf command:

      • sudo dnf install --enablerepo=elasticsearch filebeat

      Next you’ll need to configure Filebeat to connect to both Elasticsearch and Kibana. Open the /etc/filebeat/filebeat.yml configuration file using vi or your preferred editor:

      • sudo vi /etc/filebeat/filebeat.yml

      Find the Kibana section of the file around line 100. Add a line after the commented out #host: "localhost:5601" line that points to your Kibana instance’s private IP address and port:


      . . .
      # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
      # This requires a Kibana endpoint configuration.
        # Kibana Host
        # Scheme and port can be left out and will be set to the default (http and 5601)
        # In case you specify and additional path, the scheme is required: http://localhost:5601/path
        # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
        #host: "localhost:5601"
        host: "your_private_ip:5601"
      . . .

      This change will ensure that Filebeat can connect to Kibana in order to create the various SIEM indices, dashboards, and processing pipelines in Elasticsearch to handle your Suricata logs.

      Next, find the Elasticsearch Output section of the file around line 130 and edit the hosts, username, and password settings to match the values for your Elasticsearch server:

        # Array of hosts to connect to.
        hosts: ["your_private_ip:9200"]
        # Protocol - either `http` (default) or `https`.
        #protocol: "https"
        # Authentication credentials - either API key or username/password.
        #api_key: "id:api_key"
        username: "elastic"
        password: "6kNbsxQGYZ2EQJiqJpgl"
      . . .

      Substitute in your Elasticsearch server’s private IP address on the hosts line. Uncomment the username field and leave it set to the elastic user. Change the password field from changeme to the password for the elastic user that you generated in the Configuring Elasticsearch Passwords section of this tutorial.

      Save and close the file when you are done editing it. Next, enable Filebeats’ built-in Suricata module with the following command:

      • sudo filebeat modules enable suricata

      Now that Filebeat is configured to connect to Elasticsearch and Kibana, with the Suricata module enabled, the next step is to load the SIEM dashboards and pipelines into Elasticsearch.

      Run the filebeat setup command. It may take a few minutes to load everything:

      Once the command finishes you should receive output like the following:


      Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead. See more: It is not possble to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat. Loaded machine learning job configurations Loaded Ingest pipelines

      If there are no errors, use the systemctl command to start Filebeat. It will begin sending events from Suricata’s eve.json log to Elasticsearch once it is running.

      • sudo systemctl start filebeat.service

      Now that you have Filebeat, Kibana, and Elasticsearch configured to process your Suricata logs, the last step in this tutorial is to connect to Kibana and explore the SIEM dashboards.

      Step 5 — Navigating Kibana’s SIEM Dashboards

      Kibana is the graphical component of the Elastic stack. You will use Kibana with your browser to explore Suricata’s event and alert data. Since you configured Kibana to only be available via your Elasticsearch server’s private IP address, you will need to use an SSH tunnel to connect to Kibana.

      Connecting to Kibana with SSH

      SSH has an option -L that lets you forward network traffic on a local port over its connection to a remote IP address and port on a server. You will use this option to forward traffic from your browser to your Kibana instance.

      On Linux, macOS, and updated versions of Windows 10 and higher, you can use the built-in SSH client to create the tunnel. You will use this command each time you want to connect to Kibana. You can close this connection at any time and then run the SSH command again to re-establish the tunnel.

      Run the following command in a terminal on your local desktop or laptop computer to create the SSH tunnel to Kibana:

      The various arguments to SSH are:

      • The -L flag forwards traffic to your local system on port 5601 to the remote server.
      • The your_private_ip:5601 portion of the command specifies the service on your Elasticsearch server where your traffic will be fowarded to. In this case that service is Kibana. Be sure to substitute your Elasticsearch server’s private IP address in place of your_private_ip.
      • The address is the public IP address that you use to connect to and administer your server. Substitute your Elasticsearch server’s public IP address in its place.
      • The -N flag instructs SSH to not run a command like an interactive /bin/bash shell, and instead just hold the connection open. It is generally used when forwarding ports like in this example.

      If you would like to close the tunnel at any time, press CTRL+C.

      On Windows your terminal should resemble the following screenshot:

      Note: You may be prompted to enter a password if you are not using an SSH key. Type or paste it into the prompt and press ENTER or RETURN.

      Screenshot of Windows Command Prompt Showing SSH Command to Port Forward to Kibana

      On macOS and Linux your terminal will be similar to the following screenshot:

      Screenshot of Windows Command Prompt Showing SSH Command to Port Forward to Kibana

      Once you have connected to your Elasticsearch server over SSH with the port forward in place, open your browser and visit You will be redirected to Kibana’s login page:

      Screenshot of a Browser on Kibana's Login Page

      If your browser cannot connect to Kibana you will receive a message like the following in your terminal:


      channel 3: open failed: connect failed: No route to host

      This error indicates that your SSH tunnel is unable to reach the Kibana service on your server. Ensure that you have specified the correct private IP address for your Elasticsearch server and reload the page in your browser.

      Log in to your Kibana server using elastic for the Username, and the password that you copied earlier in this tutorial for the user.

      Browsing Kibana SIEM Dashboards

      Once you are logged into Kibana you can explore the Suricata dashboards that Filebeat configured for you.

      In the search field at the top of the Kibana Welcome page, input the search terms type:dashboard suricata. This search will return two results: the Suricata Events and Suricata Alerts dashboards per the following screenshot:

      Screenshot of a Browser Using Kibana's Global Search Box to Locate Suricata Dashboards

      Click the [Filebeat Suricata] Events Overview result to visit the Kibana dashboard that shows an overview of all logged Suricata events:

      Screenshot of a Browser on Kibana's Suricata Events Dashboard

      To visit the Suricata Alerts dashboard, repeat the search or click the Alerts link that is included in the Events dashboard. Your page should resemble the following screenshot:

      Screenshot of a Browser on Kibana's Suricata Alerts Dashboard

      If you would like to inspect the events and alerts that each dashboard displays, scroll to the bottom of the page where you will find a table that lists each event and alert. You can expand each entry to view the original log entry from Suricata, and examine in detail the various fields like source and destination IPs for an alert, the attack type, Suricata signature ID, and others.

      Kibana also has a built-in set of Security dashboards that you can access using the menu on the left side of the browser window. Navigate to the Network dashboard for an overview of events displayed on a map, as well as aggregate data about events on your network. Your dashboard should resemble the following screenshot:

      Screenshot of a Browser on Kibana's Security -> Network Dashboard

      You can scroll to the bottom of the Network dashboard for a table that lists all of the events that match your specified search timeframe. You can also examine each event in detail, or select an event to generate a Kibana timeline, that you can then use to investigate specific traffic flows, alerts, or community IDs.


      In this tutorial you installed and configured Elasticsearch and Kibana on a standalone server. You configured both tools to be available on a private IP address. You also configured Elasticsearch and Kibana’s authentication settings using the xpack security module that is included with each tool.

      After completing the Elasticsearch and Kibana configuration steps, you also installed and configured Filebeat on your Suricata server. You used Filebeat to populate Kibana’s dashboards and start sending Suricata logs to Elasticsearch.

      Finally, you created an SSH tunnel to your Elasticsearch server and logged into Kibana. You located the new Suricata Events and Alerts dashboards, as well as the Network dashboard.

      The last tutorial in this series will guide you through using Kibana’s SIEM functionality to process your Suricata alerts. In it you will explore how to create cases to track specific alerts, timelines to correlate network flows, and rules to match specific Suricata events that you would like to track or analyze in more detail.

      Source link