One place for hosting & domains

      How To Configure HAProxy Logging with Rsyslog on Rocky Linux 8 [Quickstart]


      Introduction

      HAProxy, which stands for High Availability Proxy, is a widely used TCP and HTTP-based proxy server that runs on Linux, Solaris, and FreeBSD. It is used to load balance applications by distributing requests between multiple servers, and to ensure that applications are highly available for users.

      By default on many Rocky Linux 8 installations, HAProxy is not configured to write its log output to a file. This quickstart tutorial will explain how to configure HAProxy logging with Rsyslog by using a Unix domain socket for reliability, speed, and security.

      Prerequisites

      To complete this tutorial, you will need a server running Rocky Linux 8. This server should have a non-root user with administrative privileges. To set this up, follow the Initial Server Setup guide for Rocky Linux 8 tutorial.

      You can also use an interactive terminal that is embedded on this page to experiment with HAProxy and rsyslog logging in this tutorial. Click the following Launch an Interactive Terminal! button to get started.

      Launch an Interactive Terminal!

      Step 1 — Installing and Enabling HAProxy

      To install HAProxy, run the following dnf command:

      When you are prompted Is this ok [y/N]: enter y and press RETURN to install the package.

      Once you have installed HAProxy, enable and start it using the systemctl command:

      • sudo systemctl enable haproxy.service

      You should receive the following output:

      Output

      Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

      With the HAProxy service now enabled, you can start it up to ensure that it runs with a default configuration on your server:

      • sudo systemctl start haproxy.service

      Next examine HAProxy’s status to make sure it is running:

      • sudo systemctl status haproxy.service

      You should receive output like the following. Note the highlighted active (running) portion of the output. If your server shows the same highlighted section then HAProxy is running correctly on your server and you can proceed with configuring logging.

      Output

      ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-09-09 21:16:39 UTC; 4min 39s ago Process: 21340 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS) Main PID: 21341 (haproxy) Tasks: 2 (limit: 2881) Memory: 2.7M CGroup: /system.slice/haproxy.service ├─21341 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid . . .

      If your output is different, or the status shows something like Active: failed, then follow the troubleshooting steps in the How to Troubleshoot Common HAProxy Errors tutorial to determine what is preventing HAProxy from starting correctly.

      Once you have confirmed that HAProxy is enabled and running, you can continue to the next step, which is configuring HAProxy’s logging directives.

      Step 2 — Configuring HAProxy Logging Directives

      To configure HAProxy’s logging directives, open /etc/haproxy/haproxy.cfg in vi or your preferred editor:

      • sudo vi /etc/haproxy/haproxy.cfg

      Press i to switch to INSERT mode, then find the line log 127.0.0.1 local2 and comment it out by adding a # character to the beginning of the line, as highlighted in the following example:

      /etc/haproxy/haproxy.cfg

      . . .
      # 2) configure local2 events to go to the /var/log/haproxy.log
      #   file. A line like the following can be added to
      #   /etc/sysconfig/syslog
      #
      #    local2.*                       /var/log/haproxy.log
      #
          #log         127.0.0.1 local2
      
          chroot      /var/lib/haproxy
          pidfile     /var/run/haproxy.pid
      . . .
      

      Now add a line directly after the commented out line with the following contents:

          log         /dev/log local0
      

      The entire section of /etc/haproxy/haproxy.cfg that you edited should contain the following lines:

      /etc/haproxy/haproxy.cfg

      . . .
      #    local2.*                       /var/log/haproxy.log
      #
           #log         127.0.0.1 local2
           log         /dev/log local0
      
           chroot      /var/lib/haproxy
           pidfile     /var/run/haproxy.pid
      . . .
      

      The chroot line is important, because it restricts the HAProxy process to accessing files in the /var/lib/haproxy directory only. The log /dev/log local0 line will create a file inside that directory that Rsyslog will use to collect log entries from.

      Once you are finished editing the log lines in /etc/haproxy/haproxy.cfg, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

      The last step that you need to complete in this section is to create the /var/lib/haproxy/dev directory since it does not exist by default.

      Create the directory using the mkdir command and then restart HAProxy:

      • sudo mkdir /var/lib/haproxy/dev
      • sudo systemctl restart haproxy.service

      You have now configured HAProxy to send its logs to a Unix domain socket that resides in /var/lib/haproxy/dev/log. In the next step, you will configure Rsyslog to create and access the socket.

      Step 3 — Configuring Rsyslog to Collect HAProxy Logs

      Rsyslog’s default configuration on Rocky Linux 8 does not handle HAProxy logs. To collect logs from the HAProxy service, open a new file /etc/rsyslog.d/99-haproxy.conf using vi or your preferred editor:

      • sudo vi /etc/rsyslog.d/99-haproxy.conf

      Press i to switch to INSERT mode, then paste the following lines into the file:

      /etc/rsyslog.d/99-haproxy.conf

      $AddUnixListenSocket /var/lib/haproxy/dev/log
      
      # Send HAProxy messages to a dedicated logfile
      :programname, startswith, "haproxy" {
        /var/log/haproxy.log
        stop
      }
      

      The $AddUnixListenSocket directive tells Rsyslog to create a Unix domain socket in the specified location, in this case /var/lib/haproxy/dev/log. The :programname, startswith, "haproxy" section specifies the file where Rsyslog will write the log entries to that it collects from the socket.

      Once you are finished editing /etc/rsyslog.d/99-haproxy.conf, save and close the file by pressing ESC, typing :wq, and pressing ENTER.

      You have now configured Rsyslog to read log entries from the Unix domain socket in /var/lib/haproxy/dev/log and write them to a log file in /var/log/haproxy.log.

      However, before restarting Rsyslog you will need to determine if SELinux is enforcing access control on your Rocky Linux 8 system.

      To check SELinux’s current policy, run the following:

      You will receive one of the following outputs:

      • Enforcing - In this mode, SELinux is enforcing access controls on your system. You will need to complete the following optional Step 4 — Configuring SELinux section.
      • Permissive - In this case, SELinux logs all access attempts to its log file, but does not enforce access controls on your system.
      • Disabled - If SELinux is disabled, then it is not logging or enforcing any access control policies on your system.

      If the getenforce command returned either Permissive or Disabled, then you can restart Rsyslog with the following command:

      • sudo systemctl restart rsyslog

      Once you restart Rsyslog, you will be able to view logs in the /var/log/haproxy.log file that you configured in /etc/rsyslog.d/99-haproxy.conf. Proceed to Step 5 — Testing HAProxy Logging to make sure that everything is working as expected.

      Otherwise, if your system is running SELinux in Enforcing mode, then the next section of this tutorial explains how to add a module to allow Rsyslog and HAProxy to communicate with each other over their shared Unix domain socket.

      Step 4 — (Optional) Configuring SELinux

      If your Rocky Linux 8 system is configured with SELinux in Enforcing mode, then you will need to allow Rsyslog access to HAProxy’s chroot directory. Allowing this access will let Rsyslog create the Unix domain socket that HAproxy will send its logs to.

      If you are using the Interactive Terminal to follow along with this tutorial, the environment does not support enabling SELinux. You can still experiment with creating and compiling the module, but you will not be able to load the module.

      If you are not familiar with SELinux, this tutorial series An Introduction to SELinux on CentOS 7 will help you learn how to manage and interact with SELinux. Although it is written for CentOS 7, the principles and commands in the series are equally applicable to Rocky Linux 8.

      To enable Rsyslog and HAProxy access to their shared socket, the first task is to create a Type Enforcement policy file. Open a new file called rsyslog-haproxy.te in vi or your preferred editor:

      Press i to switch to INSERT mode, then paste the following lines into the file:

      rsyslog-haproxy.te

      module rsyslog-haproxy 1.0;
      
      require {
          type syslogd_t;
          type haproxy_var_lib_t;
          class dir { add_name remove_name search write };
          class sock_file { create setattr unlink };
      }
      
      #============= syslogd_t ==============
      allow syslogd_t haproxy_var_lib_t:dir { add_name remove_name search write };
      allow syslogd_t haproxy_var_lib_t:sock_file { create setattr unlink };
      

      The first line defines the module name and version. The require portion tells the SELinux module loader about the types and classes that are required for the policy to be loaded as a module. The last two lines are the rules that allow Rsyslog access to HAProxy’s chroot and socket file respectively.

      When you are done editing the file, save and close it by pressing ESC, typing :wq, and pressing ENTER.

      Next, run the following command to install the checkpolicy package, which contains the checkmodule utility that you will use to turn the Type Enforcement file into an SELinux module.

      • sudo dnf install checkpolicy

      Now that you have the checkmodule tool installed, the next step is to compile the module and then load it into SELinux. Run the following to compile the Type Enforcement file into an SELinux module:

      • checkmodule -M -m rsyslog-haproxy.te -o rsyslog-haproxy.mod

      Next, run semodule_package to generate a complete policy package that SELinux can load into the Linux kernel:

      • semodule_package -o rsyslog-haproxy.pp -m rsyslog-haproxy.mod

      The final step is to load the package that you generated into the Linux kernel using the semodule command:

      • sudo semodule -i rsyslog-haproxy.pp

      Adding the module may take a few seconds. Once the command completes you can confirm that the module is loaded into the kernel by running the semodule command:

      • sudo semodule -l |grep rsyslog-haproxy

      You should receive output like the following:

      Output

      rsyslog-haproxy

      Once the module is loaded you can restart Rsyslog with the following command:

      • sudo systemctl restart rsyslog

      You have now defined, compiled, and loaded an SELinux policy that will allow HAProxy and Rsyslog to communicate over their shared socket.

      In the next step you will test that everything works by making an HTTP request to HAProxy and examining its new log file.

      Step 5 — Testing HAProxy Logging

      Now that you have configured HAProxy, Rsyslog, and optionally SELinux, you can test that logging to /var/log/haproxy.log is working.

      By default the haproxy package ships with a configuration file that creates an HTTP listener socket on port 5000. The configuration points to a non-existent backend server, so any request to the port will result in an HTTP 503 error.

      To check for a 503 error in your /var/log/haproxy.log file, first generate an HTTP request using curl like this:

      • curl -si http://127.0.0.1:5000

      You should receive output like the following:

      Output

      HTTP/1.0 503 Service Unavailable Cache-Control: no-cache Connection: close Content-Type: text/html <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html>

      Now examine /var/log/haproxy.log for any HTTP 503 responses using the grep command:

      • sudo grep -E 'NOSRV.+503' /var/log/haproxy.log

      Note: The NOSRV.+503 portion of the command is a regular expression. This tutorial on Using Grep & Regular Expressions to Search for Text Patterns in Linux
      goes into more depth on using grep and regular expressions.

      You should receive a line (or multiple lines) like the following:

      Output

      Sep 9 21:32:22 rocky-s-1vcpu-1gb-nyc3-01 haproxy[4451]: 127.0.0.1:56024 [9/Sep/2020:21:32:22.098] main app/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

      This line corresponds to the curl request that you made, which means that Rsyslog and HAProxy are configured to use their shared socket correctly.

      Conclusion

      In this quickstart tutorial, you configured HAProxy to log to a Unix domain socket. You also set up Rsyslog to create and read from the socket so that the two programs can communicate with each other without opening up any TCP/IP ports on your system. Finally, you optionally compiled, packaged, and loaded an SELinux policy to allow Rsyslog and HAProxy shared access to their socket.



      Source link


      Leave a Comment